Jump to content


ebuild is sad to announce its closure - it has become too time and resource intensive to develop, manage and maintain.

However, ebuild will remain on-line in archive mode (ie no posting facilties) for several weeks so that users can use it as an information resource.

Linux Shell Problem


  • Please log in to reply
16 replies to this topic

#1 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 02 December 2015 - 09:37 PM

Right, have wasted ages on this problem, but can't understand why it does not work rigth.

I trying to log my internet connection as it frequently drops out.

So I found this bit of shell script and it works fine when I paste it into my Raspberry Pi command line. I have even tested it by pulling out the phone lead, it does what it is meant to.

#!/bin/bash

while true
do
wget -q --tries=10 --timeout=20 -O - http://google.com > /dev/null
if [[ $? -eq 0 ]]; then
echo $(date) "1" | tee -a connection.csv
else
echo $(date) "0" | tee -a connection.csv
fi
sleep 5
done


So that was good, so saved it as a file, called connection.sh. After half a day head scratching I found out that the dos text format and the unix text format are not identical. Sorted that out with:

sed -i 's/\r$//' connection.sh

It now starts, runs and seems okay. I even have it starting at boot up.
But, when I check the log file I am getting a 0 instead of a 1 when I am connected. I pulled the phone lead, and it still gives a 0.
I downloaded a webpage with wget, it worked fine.

So any ideas what is going on?

Edited by SteamyTea, 09 December 2015 - 06:06 PM.


#2 Alphonsox

Alphonsox

    Advanced Member

  • Members
  • PipPipPipPip
  • 1,495 posts
  • LocationCounty Down, NI and Forest of Dean, England

Posted 02 December 2015 - 09:47 PM

I haven't used Bash for some time but the semi-colon in the if then statement looks suspect

if [[ $? -eq 0 ]]; then

EDIT

No, that's not it,

Tried running on my Pi and get :-

Wed Dec 2 21:52:10 UTC 2015 1
Wed Dec 2 21:52:15 UTC 2015 1
Wed Dec 2 21:53:52 UTC 2015 1
Wed Dec 2 21:53:58 UTC 2015 1
Wed Dec 2 21:54:04 UTC 2015 1

So seems to be OK

Edited by Alphonsox, 02 December 2015 - 09:55 PM.


#3 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 02 December 2015 - 09:50 PM

Should I change it to a colon?

Nope, that don't work at all.

Edited by SteamyTea, 02 December 2015 - 09:52 PM.


#4 DamonHD

DamonHD

    Advanced Member

  • Members
  • PipPipPipPip
  • 1,046 posts
  • LocationLondon, UK

Posted 02 December 2015 - 09:54 PM

I would put the wget directly in the if condition, but anyhow... I also try to stick to /bin/sh for anything with significant conditionals and other non-interactive work: I regard bash as horribly bloated and over-complex.

I use the tool dos2unix often to do conversion between DOS and UNIX text files.

Realise that I didn't have it on my current server but it is as simple as this to install on the RPi:

apt-get install dos2unix

Note that also brings in commands such as:

unix2dos unix2mac

Rgds

Damon

#5 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 02 December 2015 - 10:01 PM

Alphonsox

Did you just run it from the command line?
I don't have a problem when I do it that way, it is when I run it from the file I made.


DamonHD

I shall try the dos2unix.
I hope the instructions are easy to follow.


Could it be a problem with using upstart. Must I make sure that it is using bash or sh in both?

#6 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 02 December 2015 - 10:03 PM

Ah

I seem to have answered my own question.

I had sh in upstart and bash in the shebang.
Bit like asking for a fag in America, in the 1970's.

Seems to be OK now.

Thanks all, much appreciated.

Edited by SteamyTea, 02 December 2015 - 10:04 PM.


#7 Alphonsox

Alphonsox

    Advanced Member

  • Members
  • PipPipPipPip
  • 1,495 posts
  • LocationCounty Down, NI and Forest of Dean, England

Posted 02 December 2015 - 10:05 PM

Nope - I cut and paste you code above into a file on the pi then executed.

#8 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 02 December 2015 - 10:35 PM

Well that was weird. I pulled the phone lead on my router to test it.
Once reconnected Firefox went all strange and said it had lost my profile.
A few minutes of mucking about with reboots and it seems back to normal now.

Just in time to hear that we are off to war again.

From my log file when I pulled the phone lead and reconnected it:

Wed Dec 2 22:06:56 GMT 2015 0
Wed Dec 2 22:07:01 GMT 2015 0
Wed Dec 2 22:07:10 GMT 2015 1
Wed Dec 2 22:07:16 GMT 2015 1
Wed Dec 2 22:07:43 GMT 2015 1
Wed Dec 2 22:07:48 GMT 2015 0
Wed Dec 2 22:07:53 GMT 2015 0


Then after a few minutes of strangeness when I reconnected the network lead to the Pi.

Wed Dec 2 22:32:09 GMT 2015 0
Wed Dec 2 22:32:14 GMT 2015 0
Wed Dec 2 22:32:19 GMT 2015 0
Wed Dec 2 22:32:24 GMT 2015 0
Wed Dec 2 22:32:29 GMT 2015 0
Wed Dec 2 22:32:35 GMT 2015 1
Wed Dec 2 22:32:41 GMT 2015 1
Wed Dec 2 22:32:46 GMT 2015 1

Would a router or my ISP, have a problem with this script as I am constantly pinging an address?

Edited by SteamyTea, 02 December 2015 - 10:36 PM.


#9 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 09 December 2015 - 05:50 PM

Right, thanks for your help earlier I managed to get files moved with FTP with a bit of python script.
Trouble is that means I am sending he user name and password at the same time, across an unencrypted connection. Not a problem for testing, but not something I want to do in the real world.

Python has a library that allows SFTP, called Paramiko.
Easy to install, seems like it should be easy to use.
Followed a few instructions but I just cant get it to transfer the files. I am pretty sure it is connecting to my ftp space OK. After that it just does not seem to recognise my local directory '/home/pi/homemonitoring/' or the file

This is what I have written:

#/usr/bin/python

import sys, paramiko

host = "xxxx.xxxx.xxxx"
port = 22
transport = paramiko.Transport((host, port))

password = "xxxx"
username = "xxxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

filepath = ('xxxx.xxxx.xxxx/home/steamytea')
localpath = open('/home/pi/homemonitoring/test.csv', 'rb')
sftp.put(localpath, filepath)





Anyone know what I am dong wrong?

Thanks in advance

#10 DamonHD

DamonHD

    Advanced Member

  • Members
  • PipPipPipPip
  • 1,046 posts
  • LocationLondon, UK

Posted 09 December 2015 - 11:04 PM

FWIW I use the sftp that works with my ssh, with a key pair, so no passwords to snoop, though if it were automated it would need one to unlock the ssh key instead.

Rgds

Damon

#11 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 10 December 2015 - 05:46 AM

Is that using rsync? or similar.

#12 DamonHD

DamonHD

    Advanced Member

  • Members
  • PipPipPipPip
  • 1,046 posts
  • LocationLondon, UK

Posted 10 December 2015 - 08:26 AM

ssh is the standard secure way to log into a remote system: all the implementations that I have used come with a sftp utility of some description that works well.

If the data is not hudely private I'd consider just exposing it via a Web server (eg Apache), possibly restricted by IP address.

Rgds

Damon

#13 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 10 December 2015 - 09:01 AM

It is only some energy data, the worse that could happen is that someone deletes it, but it is easy enough to back up from the FTP site and also store a copy on the monitor.

Just a bit annoying that I know there is a way to do it, but it just does not seem to work. The examples given are fine, for someone, but don't work for me.
Most of this is down to my lack of skills, but writing programmes does not excite me any. Given a choice I refer to do the washing up (not vacuuming though, I hate that).

#14 gravelld

gravelld

    Regular Member

  • Members
  • PipPipPip
  • 655 posts

Posted 10 December 2015 - 09:53 AM

Chuck in some printlns (or whatever the equivalent is in python) and see what is *actually* happening.

The filepath - I noticed that is relative so I assume it is relative to the user folder on the FTP server. But in the path you have "/home/..." as a sub directory of some obfuscated directory - is this intentional? Just that normally on modern Linux you have the home/ off the root, i.e. /home/

You could just use SCP. What is sftp giving you?

#15 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 10 December 2015 - 02:14 PM

Thanks, you have give me a couple of pointers to look at.

What is printlns and how do I do it?

#16 DeeJunFan

DeeJunFan

    Regular Member

  • Member Blogger
  • PipPipPip
  • 698 posts
  • LocationNewry, Co. Down

Posted 10 December 2015 - 02:35 PM

View PostSteamyTea, on 09 December 2015 - 05:50 PM, said:

Right, thanks for your help earlier I managed to get files moved with FTP with a bit of python script.
Trouble is that means I am sending he user name and password at the same time, across an unencrypted connection. Not a problem for testing, but not something I want to do in the real world.

Python has a library that allows SFTP, called Paramiko.
Easy to install, seems like it should be easy to use.
Followed a few instructions but I just cant get it to transfer the files. I am pretty sure it is connecting to my ftp space OK. After that it just does not seem to recognise my local directory '/home/pi/homemonitoring/' or the file

This is what I have written:

#/usr/bin/python

import sys, paramiko

host = "xxxx.xxxx.xxxx"
port = 22
transport = paramiko.Transport((host, port))

password = "xxxx"
username = "xxxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

filepath = ('xxxx.xxxx.xxxx/home/steamytea')
localpath = open('/home/pi/homemonitoring/test.csv', 'rb')
sftp.put(localpath, filepath)





Anyone know what I am dong wrong?

Thanks in advance

I don't think you need to open the local path. Just create the variable as you have with filepath.

eg
localpath = '/home//home/pi/homemonitoring/test.csv'

open() is a method of the paramiko library just as get() and put()
so when you call
sftp.put(localpath, filepath)

you don't need to have the open() method called in your localpath

Give that a go and see what happens.

Also you don't use () when you instantiate a variable (unless the variable value is the output of a method call) which yours shouldn't be

And i think the xxxx.xxxx.xxxx is the IP address of the server. This also doesn't need to be in the filepath as its used as part of the transport details not the relative path.

I have used paramiko a few times but don't have any examples to hand (work blocks dropbox)

Your script should look like

#/usr/bin/python

import sys, paramiko

host = "xxxx.xxxx.xxxx"
port = 22
transport = paramiko.Transport((host, port))

password = "xxxx"
username = "xxxx"
transport.connect(username = username, password = password)

sftp = paramiko.SFTPClient.from_transport(transport)

filepath = '/home/steamytea'
localpath = '/home/pi/homemonitoring/test.csv'
sftp.put(localpath, filepath)

Edited by DeeJunFan, 10 December 2015 - 02:41 PM.


#17 SteamyTea

SteamyTea

    Advanced Member

  • Members
  • PipPipPipPip
  • 4,322 posts
  • LocationCornwall

Posted 10 December 2015 - 04:09 PM

I think it is something to do with how I have written the file paths.

I am using bakop.com as the free ftp.
I use host3, so the path to there is host3.bakop.com

You can probably guess my user name, not that I think that is important as there is nothing interesting there yet, and don't think I have anonymous log in anyway.

I can log into using WinSCP using username@host3.bakop.com. I can also upload from my RPi using 'normal' FTP in python (ftplib).
In that little bit of python script I have given the file I want to upload a name (text2) using text2 = open('/home/pi/homemonitoring/text.csv', 'r')
That seem to work fine (thought I would give it a go in using paramiko, but did not seem right).

I have now tried all the combinations of path names I can think of, some with /, some without, homes, double homes, just the file name.

I get a list of errors from invalid syntax, can't find file, tupples, buffers and all sorts to things dragged up from paramiko and my bit of script.

It is really wearing me down this is. It should be simple, just type the paths and and up they go, but seems not.

When I run the script I do it as root i.e. using sudo. Could that be a problem?