03/27/17

Using pipes while running external programs in Julia

Recently I was using Julia to run ffprobe to get the length of a video file. The trouble was the ffprobe was dumping its output to stderr and I wanted to take that output and run it through grep. From a bash shell one would typically run:

ffprobe somefile.mkv 2>&1 |grep Duration

This would result in an output like

 Duration: 00:04:44.94, start: 0.000000, bitrate: 128 kb/s

This works because we used 2>&1 to redirect stderr to stdout which would in be piped to grep.

If you were try to run this in Julia

julia> run(`ffprobe somefile.mkv 2>&1 |grep Duration`)

you will get errors. Julia does not like pipes | inside the backticks command (for very sensible reasons). Instead you should be using Julia’s pipeline command. Also the redirection 2>&1 will not work. So instead, the best thing to use is and instance of Pipe. This was not in the manual. I stumbled upon it in an issue discussion on GitHub. So a good why to do what I am after is to run.

julia> p=Pipe()
Pipe(uninit => uninit, 0 bytes waiting)
 
julia> run(pipeline(`ffprobe -i  somefile.mkv`,stderr=p))

This would create a pipe object p that is then used to capture stderr after the execution of the command. Next we need to close the input end of the pipe.

julia> close(p.in)

Finally we can use the pipe with grep to filter the output.

julia> readstring(pipeline(p,`grep Duration`))
"  Duration: 00:04:44.94, start: 0.000000, bitrate: 128 kb/s\n"

We can then do a little regex magic to get the duration we are after.

julia> matchall(r"(\d{2}:\d{2}:\d{2}.\d{2})",ans)[1]
"00:04:44.94"
01/13/17

Kaperkar’s Constant

I was recently introduced to Kaperkar’s Constant.
It is quite magical. You take any four digit number A, sort the digits from highest to lowest to create a new number A^{\text{high}}, sort the digits from lowest to highest to get A^{\text{low}}, and calculate and new number A= A^{\text{high}}- A^{\text{low}}. You repeat this procedure enough times and you end up with A=6174.

I made a nifty implementation of that in Julia below.

09/29/16

Making your own persistent USB bootable Ubuntu Distro: the easy way

I had a recent Ubuntu install with customizations that I wanted to be able to put it on a USB, boot into it on another machine and be able to modify its contents by, installing new packages, editing files, etc…

PinguyBuilder to the rescue

So after a number of false starts. I stumbled upon PinguyBuilder. I followed the excellent instructions at maketecheasier, with the slight modification of checking the PinguyBuilder site for the the latest version. I used the backup option to get all the tweaks on home directory, and my none deb packages in. My Ubuntu was a derivative the standard Xubuntu 16.04 distro.

Moving the ISO to the USB

The generated iso file could then be installed usb-creator-gtk, or if are working on windows, Rufus. At this point I could nicely boot using my USB stick. I could not, however, write anything persistently on the USB. If I create some file, and then reboot, it would be gone!

Making the USB persistent

First I need to create casper-rw file that would be identified by boot-loader as space to write on. I was happy with 2GB, so this what I did by adopting the instructions on StackExchange post:

I created an empty 2GB file

dd if=/dev/zero of=casper-rw bs=1M count=2048

You can changed count to whatever size of persistent storage you want. I then had to format that space into something that Linux would be able to read

mkfs.ext4 -F casper-rw

The next step was to change /boot/grub/grub.cfg file on the USB to have the following menu-entry at the beginning.

menuentry "Xubuntu in persistent mode" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  file=/cdrom/preseed/custom.seed boot=casper persistent iso-scan/filename=${iso_path} quiet splash --
    initrd  /casper/initrd.gz
}

Finally, I modified /isolinux/isolinux.cfg to include the extra label

label persistent 
  menu label live - boot the Live System
  kernel /casper/vmlinuz
  append  file=/cdrom/preseed/custom.seed boot=casper persistent initrd=/casper/initrd.gz quiet splash --

et voilà!

09/25/16

Julia calling C: A minimal example

This blog is a “Hello World” example of Julia calling C.

We start of by at bit of C code we want to call from Julia. We write the following in calc_mean.c

double mean(double a, double b) {
  return (a+b) / 2;
}

To build the library, we need to create a Makefile

CC=gcc 
 
CFLAGS=-c -Wall -fPIC
 
SOURCES=calc_mean.c 
OBJECTS=$(SOURCES:.c=.o)
 
.c.o:
    $(CC) $(CFLAGS) $< -o $@ 
 
lib: $(OBJECTS)
    $(CC) -shared -fPIC -o libmean.so $(OBJECTS)
 
clean:
    rm *.o *.so

The option fPIC and -shared are essential for Julia to be able to resolve the function in our library. Now we are almost ready to build our library. From the bash terminal we invoke:

make lib

This will generate a libmean.so file.

In Julia we call the function in our c library by

x=ccall((:mean,"libmean"),Float64,(Float64,Float64),2.0,5.0)
println(x)
3.5

For this to work,

  • Julia must be running either on the same path where libmean.so resides,
  • the path to libmean.so is in LD_LIBRARY_PATH, or
  • the path to the library is pushed to Libdl.DL_LOAD_PATH via

push!(Libdl.DL_LOAD_PATH,"path_to_libmean.so")

P.S. Thanks to Christopher Rackauckas for tips on Julia highlighting.

08/19/16

In the wind

I stand alone!

Bellowing winds, drive me home

In agony, I cry for a dream forlorn 

To build a world that is free of form

A world where Truth can again be born

Of purest thought, of noble intention

A world so free from any contention 

Alas, my soul now must now take flight 

As darkness falls in a starless night 

I ride the winds to from whence I came 

A place were Light will always remain 

Where my dream can be reborn again 

09/15/15

Questions and speculations on Consciousness

I think, I am, I have a sense of self, and my observations shape reality. There is an apparent intentionality in my actions. I believe I possess the power rebel against existing order, tear it apart and construct a new one. I am a member of a lucky species that has attained a certain degree of consciousness.

But what is that thing we label consciousness. Is it simply and emergent phenomena of an advanced sensory system that is combined with the complex arrangement of neurons and synapses? Is it inevitable that conscious being such as humans should exist in the universe? Are we truly “At home in the Universe”, as Stuart Kauffman would put it?

Is the quality of consciousness intrinsically tied to being human? I am not so sure.

Is it a field that permeates space seeking a suitable medium to express itself? I am happier with that thought, though I have no way to prove it…yet!

We might one day set in motion a process that would create increasingly more advanced mediums of consciousness. We may one day surpass ourselves and hand the baton to nobler and superior beings. There should be joy in that; no fear or shame. It will be just then that we would have fulfilled the ultimate purpose of our species.

07/11/15

On terrorism and the hate that follows

I worry less about the damage to live and property from acts of terror, than the waves of hate that soon follow. The damage due to hate is always much worse. Hate brings out the worst in humans. It locks the mind in inescapable focus on violence. All noble ideas, and worthy aspirations fall by the wayside as survival becomes the primary preoccupation. When that happens, I sense that the terrorists have already won.

07/11/15

الكراهية أخطر من الإرهاب

أرى نفسي أقل قلقاً من الموت و الدمار الناتج من الأعمال الإرهابية بالمقارنة لموجات الكراهية التي تتبعها. الضرر و الخسارة الناتجة من الكراهية التي تولدها هذه الأعمال أشد خطورة.
الكراهية تسقم القلوب و تبرز أقبح ما في البشر.
الكراهية تسجن العقل في دائرة لا نهائية في التركيز علي العنف.
كل الأفكار النبيلة و الطموحات الشريفة تسقط عندما يكون الهم الشاغل هو البقاء.
عندما يحدث ذالك أشعر أن الإرهاب قد نال غايته.

07/11/15

Herodotus on Decision Making

A decision was wise, even though it led to disastrous consequences, if the evidence at hand indicated it was the best one to make; and a decision was foolish, even though it led to the happiest possible consequences, if it was unreasonable to expect those consequences.