The project aim is to make a robotic arm easy to use with devices that do not need to be worn. This system provides a simple front end that hides the complexity in the programming of a robotic arm.

It’s a system that can work with existing applications, but provides new usage scenarios.

If the user lifts only a finger, any kind of events can be generated. For example , drawing or marking specific paths to be tracked can be easily done just by moving the index finger in the space.

If a user would like a higher accuracy on the path, a tablet can be used to refine the trajectories.
The user by means of the hand has the ability to indicate the point where the robot takes an object and the destination where the robot leaves it.

Appropriate checks will be implemented to ensure the safety of users.
The main objective is to create a collection of devices and software tools allowing a simple, fast but also safe human-machine interaction.

more info:

http://www.rosariobarbagallo.com/kinect-doctor/

This software, based on the Microsoft Kinect, uses hand motion detection based on a clustering hand detection algorithm.

The coordinates of the hands are sent to an application that computes the inverse kinematics and the desired robot trajectory.
A raised finger sends an input. With the hand closed, the cursor is displayed on the screen without transmitting touch input.

Fig. 1 Working Area of Kuka.

The method is used to send the coordinates when the finger is detected, then, when the hand is closed the trajectory is considered concluded. It is necessary to keep the hand closed (first) because when the user lifts a finger, the system sends an input signal (touch).

Fig.2 Software C# Multithread “Kinect Writer”.

To improve the quality of the trajectory, interpolation and moving average has been implemented.

A system of events linked to voice commands is also integrated with multiple features. To avoid the activation of unwanted commands, the user must keep both hands closed while saying the command. All controls to be tested must have a confidence level greater than a given threshold (to be decided experimentally).

The User writes: “Ciao”

The User writes: “Rosario” (My name! )

The lexical analyzer should take as input a MINI-L program, parse it, and output the sequence of lexical tokens associated with the program.
The software takes a syntactically-correct MINI-L program (a program that can be parsed without any syntax errors), verify that it has no semantic errors, and then generate its corresponding intermediate code.
A source program in MINI-L will be translated by the compiler into an intermediate code program, and that intermediate code program will be the input to the interpreter which will take the intermediate code, execute it and give the final results.

MIL CODE

The MINI-L Language has the following features:

1. Integer scalar variables.
2. Unidimensional arrays of integers.
3. Assignment statements.
4. While loops.
5. If-then-else statements.
6. Read/Write statements.
7. Comments.

In MINI-L a comment is introduced by “- -” and goes to the end of the line. All tokens are separated by blanks, tabs, newlines, delimiters, operators or comments. A valid identifier begins with a letter and may be followed by additional letters, digits or underscores. The length of an identifier does not exceed 8 characters. Finally, MINI-L is case sensitive with all reserved words written in lower case.

Errors handled:

Syntax diagrams describing the syntax of MINI-L are given to you. Your first task in this assignment
is to develop a context-free grammar from the syntax diagrams. The parser will get the necessary information about the tokens from the lexical analyzer.

MIN CODE

Each line in the MIN file

1. contains at most one MIL instruction

2. Each line is at most 254 characters long

3. All variables are defined before they are used

Code Source Example:

program test_er;

n,x: integer;

d: integer;

a,b : array(1000) of integer;

c : array(10) of integer;

beginprogram — main program

– test op

n:= x+10*20/n-d%40+(2+20)*a(2);

– test if

if x>0 and n==d or a(10)>=c(n)

then d:=d*d*d*d; endif;

–test =

d:=11; d:=x; a(10):=2; a(10):=c(2);

– test if-else

if x+20/n>x-1*n

then n:=10+d;

if a(10)>=c(n)

then n:=d*x;

else n:=20*n;

endif;

endif;

– test while

while x<-10 or not c(2)!=0 loop

x:=x+1;

endloop;

– test read

read n;

read n,x,a(10);

endprogram

Output Code:

.n
.x
.d
.[] a, 1000
.[] b, 1000
.[] c, 10
.t0
.t1
.t2
.t3
.t4
.t5
.t6
.t7
.t8
.p0
.p1
.p2
.t9
.t10
.p3
.p4
.p5
.t11
.t12
.t13
.t14
.t15
.t16
.t17
.t18
.p6
.p7
.t19
.t20
.t21
.p8
.p9
.t22
.t23
.t24
.p10
.t25
.p11
.p12
.p13
.p14
.p15
.t26
: START
/ t0, 20,n
* t1,10,t0
% t2,d,40
– t3,t1,t2
+ t4, x,t3
+ t5,2,20
=[] t6,a,2
* t7,t5,t6
+ t8,t4,t7
= n,t8
> p0,x,0
== p1, n,d
&& p2,p0,p1
=[] t9,a,10
=[] t10,c,n
>= p3,t9,t10
|| p4,p2,p3
== p5,p4,0
?:= L000 ,p5
* t11, d,d
* t12, t11,d
* t13, t12,d
= d,t13
: L000
= d,11
= d,x
[]= a,10,2
=[] t14,c,2
[]= a,10,t14
/ t15, 20,n
+ t16, x,t15
* t17,1,n
– t18, x,t17
> p6,t16,t18
== p7,p6,0
?:= L001 ,p7
+ t19,10,d
= n,t19
=[] t20,a,10
=[] t21,c,n
>= p8,t20,t21
== p9,p8,0
?:= L002 ,p9
* t22, d,x
= n,t22
:= L003
: L002
* t23, 20,n
= n,t23
: L003
: L001
: L004
– t24,10
< p10, x,t24
=[] t25,c,2
!= p11, t25,0
! p12, p11
|| p14,p10,p12
== p15,p14,0
?:= L005 ,p15
+ t26,x,1
= x,t26
:= L004
: L005
.< n
.< n
.< x
.[]< a,10
.> n
.> x
.[]> a,10

Scheme TUIO SKELETON

 TUIO SERVER sends data                            TUIO CLIENT receives data

How It Works

Thanks to the features  of the Tuio protocol, it has been implemented a client-server system for the transmission of the skeleton data  kinect “real time”. The results have been excellent opening new scenarios for the TUIO protocol  use.The data transmission is fast and efficiently, the protocol TUIO is redundant and provides reliability

On the left there is the server                   On the right the Client

The Tuio Server divides the points of the skeleton in 5 groups and each group is assigned an id.

BodySegment(0,skeletons[k].Joints[JointType.ShoulderLeft],skeletons[k].Joints[JointType.ElbowLeft], skeletons[k].Joints[JointType.WristLeft], skeletons[k].Joints[JointType.HandLeft]);

BodySegment(10, skeletons[k].Joints[JointType.ShoulderRight], skeletons[k].Joints[JointType.ElbowRight], skeletons[k].Joints[JointType.WristRight], skeletons[k].Joints[JointType.HandRight]);

BodySegment(30,skeletons[k].Joints[JointType.HipLeft],skeletons[k].Joints[JointType.KneeLeft], skeletons[k].Joints[JointType.AnkleLeft], skeletons[k].Joints[JointType.FootLeft]);

BodySegment(50,skeletons[k].Joints[JointType.HipRight],skeletons[k].Joints[JointType.KneeRight], skeletons[k].Joints[JointType.AnkleRight], skeletons[k].Joints[JointType.FootRight]);

BodySegment(80,skeletons[k].Joints[JointType.Spine],skeletons[k].Joints[JointType.ShoulderCenter], skeletons[k].Joints[JointType.Head], skeletons[k].Joints[JointType.ShoulderCenter]);

The server sends add the TuioCursor  (id, [joint.x, joint.y]) and commit the Frame:

_server.AddTuioCursor(id, (s_tmp));

_server.CommitFrame();

The client, thanks to the id, understand the type of point received.

if (tcur.getSessionID() == xxx){

…

…

}

The client graphics without problems the skeleton using five polyline.

Detail of the Client which displays the received messages in the console and the graphics rendering of points received

You can see a Demo of my application here:

OSC
Open Sound Control is a transmission protocol that allows musical instruments, computers and other devices to exchange multimedia creators that make it easy to “music performance data” in real time through a simple internal network (TCP / IP, Ethernet) or the Internet.

OSC works on high-speed network (broadband network speed), to exchange more data compared to MIDI, also giving more flexibility to the type of data available, this implies the possibility of a higher level of interaction between machines / software. The protocol for data exchange network used is the UDP.

A software can choose any number of ports, only limited by bandwidth, and also to send data to receive, and each signal has a path that can be assigned to an associated URL.

This technology allows to send different types of data including 32-bit packages, provides bandwidth for the exchange of data much larger, 1024 simultaneous messages against 128 of the MIDI, low latency and 32-bit data resolution.

TUIO PROTOCOL

The TUIO protocol is encoded using the format Open Sound Control, which provides an effective method of encoding binary data for transmission. The default transport method for the TUIO protocol is the encapsulation of binary data in OSC message within UDP packets sent to port 3333. The TUIO protocol defines two main classes of messages: SET messages and ALIVE messages. SET messages are used to communicate information about an object’s state such as position, orientation, and other recognized states. ALIVE messages indicate the current set of objects present on the surface using a list of unique Session IDs. To avoid possible errors evolving out of packet loss, no explicit ADD or REMOVE messages are included in the TUIO protocol. The receiver deduces object lifetimes by examining the difference between sequential ALIVE messages. In addition to SET and ALIVE messages, FSEQ messages are defined to uniquely tag each update step with a unique frame sequence ID. An optional SOURCE message identifies the TUIO source in order to allow source multiplexing on the client side.

Efficiency & Reliability

In order to provide low latency communication our implementation of the TUIO protocol uses UDP transport. When using UDP the possibility exists that some packets will be lost. For efficiency reasons set messages are packed into a bundle to completely use the space provided by a UDP packet. Each bundle also includes a redundant alive message to allow for the possibility of packet loss. For larger object sets, a bundle with a series of set messages, each including an alive message, are transmitted. When the surface is quiescent, alive messages are sent at a fixed rate dependent on the channel quality, for example once every second, to ensure that the receiver eventually acquires a consistent view of the set of alive objects. The state of each alive but unchanged object is periodically resent with additional set messages. This redundant information is resent at a lower rate, and includes only a subset of the unchanged objects at each update. The subset is continuously cycled so that each object is periodically addressed. Finally, each packet is marked with a frame sequence ID (fseq) message: an incrementing number that is the same for all packets containing data acquired at the same time. This allows the client to maintain consistency by identifying and dropping out-of-order packets.

 ——————————————————————————————————-

References:

http://www.tuio.org/

http://opensoundcontrol.org/

This article describes both the implementation of an input provider, based on the Kinect for Windows SDK v1.0 and the algorithm of hand detection. The input provider sends the coordinates to the windows driver using a clustering hand detection algorithm, which analyzes data in depth at a distance between 800 mm and 1100 mm. The driver sends the data to a virtual Human Interface Driver that translates the “touch messages”.

This input providers allows us to use any application (past and future) that works on touch devices. The developer can use this type of software to do also different applications like for example home automation applications.

The use of the TUIO protocol guarantees not only low latency but also a certain robustness because it includes redundant information to correct any lost data packets.
The possibility of transmitting data over the network allows us to use the kinect without a direct connection to the PC where the final application is installed. This way the control can be distributed over multiple machines. The Kinect has made faster processing, and it has combined the use of gestures and voice control that extends the usage scenario.

Project Information URL: http://www.barbagallogiuseppe.com/Prof/MultiTouchDriverKinect.pdf

I and a my friend have made an Android application for listening to music, with the use of technology QRCode, WebService (https) to retrieve the information for each user (login -logout) and the songs.

You Tube video:

I and a my friend have been developed a system for filling bottles of water, the system is controlled by a Siemens PLC.
We have made: position sensors, water level sensor, engine sensor and touch screen interface, programmed PIC16F.

System Architecture:

Scheme Sensor:

Scheme Engine:

You Tube video:

How it works

The router receives the association request from the device and starts a timer when the timer expires, the router computes the new channel using the random function and makes the jump.
During the execution of the timer, the router receives data from the device.
The device, received confirmation from the Association Router, start the timer.
When the timer expires, the device calculates the new channel and makes the jump. During the execution of the timer device sends the data only after the data generation timer has expired.

JAVA SW

I also have developed a software in Java that processes data and statistics:

1. Histogram Packets Sent, Received and Lost.
2. Delay maximum, average, minimum and standard deviation.
3. Histogram of the delay intervals, with the possibility to select the desired interval interface.
4. Histogram Average Throughput (bits \ second).
5. Throughput Graph XY (horizontal axis – second , order – average throughput (bit \ s).
6. Chart Total Package and Delay for all devices on the network.
7. Number of jumps.