Thursday, March 9, 2017

Using dhclient to automatically change IP addresses of Raspberry Pi on start-up

So, in my previous post I dealt with changing IP addresses on my Pi cluster by scp'ing the new IP addresses after the Pi had started up and established a network, and creating an ssh/config file from that info.

This is fine for most occasions. But I have one instance set-up to run zookeeper. I'd have to go around and change the config files on all my other Pi's. Ansible could cure this, and I'll get there one of these days. But I'm not there today. And actually - I think I'll use the script I wrote previously to update ansibles' hosts file. I just now thought of that.

Anyway - I want the zookeeper Pi to always have the same address.

I combined this question on with what I learned about creating systemd start-up jobs to request the same ip address from my hotspot.

The gist of the answer is add send dhcp-request-address 192.168.0.XXX to /etc/dhcp/dhclient.conf.

I then created a short executable script in /usr/bin called
$cat /usr/bin/
#! /bin/bash

dhclient -r -v && dhclient -4 -d -v -cf /etc/dhcp/dhclient.conf wlan0
Then I created the service file:
# cat /etc/systemd/system/request_ip.service
Description=Change ip to 131 for myProject


I installed request_ip.service in /etc/systemd/system, and enabled it with systemctl enable request_ip.service.  Checking syslog, I know it ran. I'll update this post if I run into problems.

One way to deal with changing IP addresses on my raspberry Pi cluster

One of the annoyances I have with my router (it's actually a hotspot) is that every time I start up my Pi cluster, they all have different IP addresses than they had the day before.

So - I have two ways to deal with this.

The first way - I scp the IP addresses as a file (after the pi's start up) to my work computer, then generate an ssh_config file. The second way, I request the same IP from my hotspot. All my Pi's are running Debian Jessie with systemd.

The first way: 

scp (or you could email it) the IP address after networking has started up.

The links:

Running Services After the Network is up
[Solved] SystemD, how to start a service after network is up.


To do this with systemd, I had to create the script to scp the ip, make sure it was executable and then copied to /usr/bin/.

Then, I had to create a systemd service file, copy that to /etc/systemd/system/ and enable the service.

The Details:

The script to scp the IP address:

#! /bin/bash

IP_ADDR=`ifconfig wlan0 | grep 'inet addr:' | cut -f 12 -d ' ' | cut -c 6-`
echo $IP_ADDR > /tmp/$HOSTNAME

scp -i /home/panchod/.ssh/id_rsa -o StrictHostKeyChecking=no /tmp/$HOSTNAME panchod@
You'll undoubtedly notice -o StrictHostKeyChecking=no. This is to avoid the nastiness of the host not being in the known_hosts file.

The service file:

$ cat send_ip.service
Description=Send ip address to my Toshiba on startup


And then I wrote a script to install everything:
$ cat
#! /bin/bash

sudo cp -iv /usr/bin
sudo cp -iv send_ip.service /etc/systemd/system/send_ip.service
sudo systemctl enable send_ip.service

I made executable. Put them all in the same directory. Tar'd and zip'd the directory. Then, just scp'd the tarball to each pi, untar'd the tarball and ran Done and done. I should have used ansible - but I'm not there yet.

On my work machine - it's kind of kooky how I have it set up, so it's probably not what you'd do. Because I use ssh for work, and therefore can't mess with .ssh/config (because it's shared) I'll tell you what I do instead.

I have an alias set up for my local lan and use a separate config file.
alias lssh='ssh -F ~/lan/ssh_config'
ssh_config looks like this:
$ cat ~/lan/ssh_config
Host localCluster.*
  User root
  Port 22
  IdentityFile /home/panchod/.ssh/id_rsa-new_pi_net

Host localCluster.bpi-iot-ros-ai
Host localCluster.raspberry-pi-1
Host localCluster.raspberry-pi-2
Host localCluster.raspberry-pi-3
Host localCluster.raspberry-pi-4
Host localCluster.raspberry-pi-6
 and I have bash completion set-up for it:
#                       LAN SSH Autocomplete

    comp_lssh_hosts=`cat ~/lan/ssh_config| \
        grep "^Host " | \
        awk '{print $2}' | grep -v "\*"
    COMPREPLY=( $(compgen -W "${comp_lssh_hosts}" -- $cur) )
    return 0
complete -F _complete_lssh_hosts lssh
So - when I type lssh <tab> it starts off with localCluster. I can then hit <r><tab> for raspberry-pi and it fills in the rest, then just pick which one I want.

I also had to make a script to generate the ssh_config file. This is a little more complicated than it needs to be, but there was a reason I did it this way.

First - I have all the pi stuff in ~/lan/. As you can see above - the IP addresses are in ~/lan/ip_addresses. I put the header of ssh_config in ~/lan and called it ssh_config_preface.txt. It looks like this:
 Host localCluster.*
  User root
  Port 22
  IdentityFile /home/panchod/.ssh/id_rsa-new_pi_net

The script to generate the config that you see above:
#! /bin/bash


cat /home/panchod/lan/ssh_config_preface.txt > $SSH_FILE

if grep localCluster $CLUSTER_FILE ;
    echo "Cluster exists in $CLUSTER_FILE - replacing it"
    sed -i 's/localCluster.*/localCluster    /g' $CLUSTER_FILE
    echo "Cluster doesn't exists creating it"
    echo "localCluster        " >> $CLUSTER_FILE

for i in /home/panchod/lan/ip_addresses/* ; do

    NAME=`basename $i`
    #echo $NAME
    IP=`cat $i`
    #echo $IP

    echo "Host localCluster.$NAME
    HostName $IP" >> $SSH_FILE
     sed -i "s/^localCluster.*$/& $IP/g" $CLUSTER_FILE

echo $SSH_FILE
echo; echo

I also use cssh, so I have an alias for that as well: alias lcssh='cssh -c /home/panchod/lan/cluster'.

I should add a test to ensure that the files exist, but this isn't production,and I only wrote them for me, so if they fail - no one to blame but myself.

[Edit: changed the location of .ssh_config for consistency]

Saturday, January 28, 2017

HOW-TO use Kafka to stream Postgresql traffic with bottledwater_pg on a Raspberry Pi 3

If you want to stream changes to your Postgresql database to Kafka on your Raspberry Pi 3 - this is how I did it. I'm sure there are other ways to accomplish this. All I want to do here is to show you how I managed it. I'll preface my comments with a hash '#'.

First: The links.
Apache Kafka
Apache Zookeeper
Apache Avro 

Here goes:
sudo apt-get update && sudo apt-get -y upgrade
sudo apt-get install oracle-java8-jdk git
I'm pretty sure you only need a jre, but I installed the jdk for a different project

Install zookeeper and start
This is optional - the confluent package has a limited version of zookeeper

Compile and install postgresql from source
The version of postgresql provided by Debian jessie is older than that required by bottledwater_pg

Adjust the values of pg_hba.conf and postgresql.conf. I used 'kafka_user' but that's up to you:
sudo -i
echo "wal_level = logical
max_wal_senders = 8
wal_keep_segments = 4
max_replication_slots = 4" >> /usr/local/pgsql/data/pg_hba.conf

echo "local   replication     kafka_user                 trust
host    replication    kafka_user   trust
host    replication    kafka_user  ::1/128        trust" >> /usr/local/pgsql/data/postgresql.conf

You're going to add that user "kafka_user" to postgres. This assumes that you followed the link above and created the database 'kafka_user':
export PATH=$PATH:/usr/local/pgsql/bin
psql -Upostgres -c "create role kafka_user with SUPERUSER LOGIN;"
createdb -Upostgres kafka_user
psql -Upostgres -c "REVOKE CONNECT ON DATABASE kafka_user FROM PUBLIC;"
psql -Upostgres -c "GRANT CONNECT ON DATABASE kafka_user TO kafka_user;"
Now you need Apache Avro, or at least the C and maybe C++ portions. I couldn't get the complete avro package to compile out-of-the-box. Originally I installed all of the pre-requisites, but ended up just installing the C and C++ portions.

Get the confluent package:

sudo unzip -d /opt
 Install bottledwate_pg:
sudo -i
cd /opt
git clone
apt-get install libsnappy-dev librdkafka-dev libcurl4-openssl-dev libpq-dev
export PKG_CONFIG_PATH=/lib/pkgconfig/
ln -s /usr/local/pgsql/bin/pg_config /usr/bin/pg_config
make install
I added the kafka_user to my Raspberry Pi because I didn't want to figure out the right postgres connection string:
sudo useradd kafka_user
Almost there. Now just follow the instructions on bottledwater_pg's github page:
# start kafka server as root (zookeeper should already be running)
cd /opt/confluent
./bin/kafka-server-start ./etc/kafka/

# start schema registry
./bin/schema-registry-start ./etc/schema-registry/

# start bottled water
## I had to do this as user kafka_user
kafka_user@raspberrypi:/opt/bottledwater-pg $ ./kafka/bottledwater --postgres=postgres://localhost
-- or --
runuser -l kafka_user -c"cd /opt/bottledwater-pg; ./kafka/bottledwater --postgres=postgres://localhost"

#create table in postgresql (this creates a topic - which should be the same name as the table). Make sure it has a primary key.
/usr/local/pgsql/bin/psql -Ukafka_user kafka_user -c "CREATE TABLE test_table_with_pk(id INT NOT NULL PRIMARY KEY, some_text VARCHAR, a_float_val FLOAT);"
# list topics
root@raspberrypi:/opt/confluent-3.0.1# bin/kafka-topics --list --zookeeper localhost:2181
Java HotSpot(TM) Server VM warning: G1 GC is disabled in this release.

#listen to topic
root@raspb32:/opt/confluent-3.0.1# ./bin/kafka-avro-console-consumer --topic table_with_pk --zookeeper localhost:2181  --property print.key=true --from-beginning

Insert some data into your table, and you can see it in the consumer:
terminal running postgres:
kafka_user=# SELECT * FROM table_with_pk;
 id |  test_text  | a_float
  1 | some text 1 |     1.8
  2 | some text 2 |     2.8
  3 | some text 3 | 2.87609
  4 | test text 4 |      12
(4 rows)

kafka_user=# INSERT INTO table_with_pk VALUES(5,'test text5', 17.98);

terminal running a kafka consumer:
root@raspb32:/opt/confluent-3.0.1# ./bin/kafka-avro-console-consumer --topic table_with_pk --zookeeper localhost:2181  --property print.key=true --from-beginning
OpenJDK Zero VM warning: G1 GC is disabled in this release.
{"id":{"int":4}}    {"id":{"int":4},"test_text":{"string":"test text 4"},"a_float":{"double":12.0}}
{"id":{"int":5}}    {"id":{"int":5},"test_text":{"string":"test text5"},"a_float":{"double":17.98}}
Please let me know if you run into any problems - or especially if there are errors in this HOW-TO.


Installing the C AVRO library

Reference: Apache Avro

git clone

cd avro
sudo apt-get install gcc cmake asciidoc source-highlight libjansson-dev
cd lang/c
mkdir build
cd build/
make test
sudo make install

# c++

sudo apt-get install libboost-filesystem-dev libboost-system-dev libboost-program-options-dev libboost-iostreams-dev g++ flex  bison  libboost-dev cmake 
cd lang/c++
cmake -G "Unix Makefiles"
make install

Install postgresql from source an Raspberry Pi 3 Debian Jessie


sudo -i
cd /usr/src/
tar xf postgresql-9.6.1.tar.bz2
cd postgresql-9.6.1
apt-get install -y libreadline-dev libperl-dev libpython-dev libxml2-dev libssl-dev
# become non-root user
./configure --with-python --with-perl --with-openssl --with-libxml --enable-debug

su -i
make install
useradd -M postgres
mkdir /usr/local/pgsql/data
chown postgres /usr/local/pgsql/data
su - postgres
/usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
/usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data > /tmp/postgres_logfile 2>&1 &

Feel free to add that path (export PATH=$PATH:/usr/local/pgsql/bin/) to your .bashrc.

Install zookeeper package

As part of the Kafka streaming project, I installed zookeeper. Here's how I did that:

Ref: Apache Zookeeper

tar xf zookeeper-3.4.9.tar.gz -C /opt/
sudo  cp  /opt/zookeeper-3.4.9/conf/zoo_sample.cfg /opt/zookeeper-3.4.9/conf/zoo.cfg

sudo /opt/zookeeper-3.4.9/bin/ start
#ensure zk is running
echo stat | nc 2181

With nothing connecting to zookeeper, the output will look something like:

Zookeeper version: 3.4.9-1757313, built on 08/23/2016 06:50 GMT

Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x0
Mode: standalone
Node count: 4