Jamie's Mad Scientist Lab

Home / Fall 2015 / He’s alive… HE’S ALIVE!!!

He’s alive… HE’S ALIVE!!!

He finally works!!! Here’s a little video of a conversation with my alien.

Extraterrestrial Artificial Intelligence Early Testing from Jamie Ruddy on Vimeo.

This is what it took to get him there:

This week so far has been completely insane. After catching a few hiccups in the P5 IDE, I moved the code for my AI extraterrestrial character to Sublime Text, but I did keep the P5 libraries. This solved issues I was having with the Event Listener and once I solved that, I was able to implement my code to search for key words and have the alien actually respond!!! The first time he talked back, I almost fell out of my chair. I must admit this was the highest high I have felt from coding since I started learning to program three months ago. BUT then he crashed.

Problem Number One:

I started with this bit of code.

function handleSpeech(event) {
print(event.results);
var res = event.results[i][0].transcript;
print(res);
// logic here to search for keywords and responds
i++;
}

First I switched the i to the word index so it wouldn’t run into problems with the variable i for my for loop. But after the alien answered one question,  the error it kept throwing was: Javascript Uncaught TypeError: Cannot read proper ‘O’ of undefined. I thought that the event.results was adding on after each and every time you speak, but maybe it’s because I was starting and stopping the recognition, it always started again at zero. Shawn Van Every caught this and had me eliminate the i that had become index and the index++. That stopped the error, but he still only answered just one question and then the screen would go black.  The console log also no longer printed that the recognition had restarted.

Problem Number Two:

If that error wasn’t causing him to stop, what was? This is what I have spent the the last three days trouble-shooting. The way the logic works for the application is when you log in, it triggers the opening video to play. Each of the alien’s responses are actually video files. The speech recognition picks up his words so I need to start and stop recognition constantly. This is why I needed to add the event listener. Each and every video has an event listener that calls the same function. When the video ended function is called, the holder still image screen picks up when the video drops out and the speech recognition restarts. Whenever he crashes, the screen goes black and the speech recognition does not kick in according to the console log. That’s why I knew somehow the video ended function was never called. My first thought was the computer was getting lost in my for loop. Here is the beginning code for it:

recognition.stop();
console.log(“Speech”);
var w = split(res, ” “);
res = ”;

for (var i = 0; i < w.length; i++) {
print(w[i]);
words.push(w[i]);
}

for (var i = 0; i < words.length; i++) {

if (words[i] == “meaning”) {
background(0);
screenSeq = 3;
meaningOfLife.play();
words = [];
w = [];
return;
} else if (words[i] == “where”) {
background(0);
screenSeq = 4;
whereAreYou.play();
words = [];
w = [];
return;

The list of else if goes on and on. In this code above, these are the things I added as I trouble-shooted. One thought was the extra words from last time were messing with it so I added the res ‘ ‘; to clear that variable. Then I added the words = []; and the w = []; to clear those, but that didn’t work. Then I thought it was because the loop ran over and over again. Since I wanted it to stop once it found a key word, I figured I could add in a return; at the end of each one. This was hugely helpful in that it sped up the results, but he still broke. At this point sometimes he answered one question, sometimes two and sometimes even three, but he always went to black eventually. It was still a problem with the event listener not knowing when the video ended.

I had the thought that the problem was the part in the draw loop. The logic is that I used a switch statement to call each scene. In the above code, you can see the screenSeq number assigned in each keyword option. The key word triggers a function that tells the video to play and display it on the screen. I tried moving this switch statement out of the draw loop, but it broke him completely so I moved it back.

function draw() {

switch(screenSeq) {
case 0:
displayScene01();
break;
case 1:
displayScene02();
break;
case 2:
displayScene03();
break; // (etc. this goes on for a while)

This in turn calls these:

function displayScene04() {
//recognition.stop();
background(0);
image(meaningOfLife, 0, 0);
// meaningOfLife.play();
console.log(meaningOfLife.currentTime);
}

function displayScene05() {
//recognition.stop();
background(0);
image(whereAreYou, 0, 0);
// whereAreYou.play();

} // etc. These go on endlessly as well.

It turns out my thinking was not that far off. After spending many more hours on this, I decided that for ICM class today, at least if I refresh him enough times, I can get him talking so that’s as good as I was going to get. Pulling an all-nighter was not going to get me to solve this code.

THE NEXT MORNING…

Introduce my hero, Dan Shiffman. I must say he’s been a hero of mine many times over when I code myself into a corner. My logic was correct. The videos in the draw loop were the problem. BUT the rest of the display scene functions needed to be called constantly in a loop. If the image of the videos wasn’t called in a loop, they wouldn’t be visible on the screen, BUT the video.play itself should only be called once. Dan spotted this and had me move the play.video part of the code into the for loop, the rest of the code including the displaying of the video stayed in the draw loop. This worked! He began to speak almost indefinitely.

I also had to fix the way I coded the answer he gave if you didn’t say a key word. There is so much more I want to add – answers I’ve created, but haven’t coded yet, a random function calling one of numerous options if he doesn’t find a key word so he doesn’t say the same thing over and over, and much, much more testing to figure out more answers I need to compose for him, BUT as Shiffman said for my final class of ICM today, quit while I’m ahead. Show him as is and do not mess with it any more for now. Since Shiffman is my hero today, I’m taking his advice.

There is much as I’ve written above I’d like to change about this interaction. Even more when I consider what user testing has already pulled up and will influence the interaction more in the future, but here is what I have built so far if you want to test him out. Because it is on the internet and not on a local server, if you play with him, every time the speech recognition starts, it will ask you permission to use the mic. I do not have a way around this yet. Also note, it will only work in the Chrome browser as of now so if you want to try it, only try it in Chrome.

http://jamieruddyitp.com/projects/ICM/alien/

I built in if it does go to black, hit the space bar key and it should fire the function to keep it going. I haven’t had to use this since I’ve recoded him, but I put it in just in case. I’m not sure if it will happen to you, but a new little bug I noticed is when you first log in, sometimes it goes to black. Not sure the cause of this yet, but if that happens refresh and start it again.

I am going to paste the messy version of my code with all of the parts either commented out or still there for trouble shooting (not effecting his functionality). This way you can see a lot of my trial and error in the coding. I plan to clean him up a bit more and see if I can make this more effective, but he works for now and as someone who has coded for only three months, I am very, very excited. He’s alive… HE’S ALIVE!!!

var screenSeq = 0;

var logo;
var title;
var videoEnd = false;
var opening;
var meaningOfLife;
var whatLookLike;
var whereAreYou;
var carbonBased;
var music;
var howLongLive;
var howOld;
var raceViolent;
var sendPicture;
var speakAnotherLanguage;
var beenHereBefore;
var favoriteFood;
var yourName;
var whatShipMade;
var randomOne;
var unspeakables;
var doYouDream;
var sex;
var words = [];
var stillImage;
//var videoPlayed = 0;

var index = 0;
var recognition;

function preload() {
logo = loadImage(‘assets/PETI2.png’);
title = loadImage(‘assets/PETItitle.png’);
stillImage = loadImage(‘assets/stillImage.jpg’)
opening = createVideo(‘assets/opening.mp4’);
opening.hide();
opening.elt.addEventListener(“ended”, videoEnded);
meaningOfLife = createVideo(‘assets/meaningOfLife.mp4’);
meaningOfLife.hide();
meaningOfLife.elt.addEventListener(“ended”, videoEnded,false);
whereAreYou = createVideo(‘assets/whereAreYou.mp4’);
whereAreYou.hide();
whereAreYou.elt.addEventListener(“ended”, videoEnded);
carbonBased = createVideo(‘assets/carbonBased.mp4’);
carbonBased.hide();
carbonBased.elt.addEventListener(“ended”, videoEnded);
music = createVideo(‘assets/music.mp4’);
music.hide();
music.elt.addEventListener(“ended”, videoEnded);
howLongLive = createVideo(‘assets/howLongLive.mp4’);
howLongLive.hide();
howLongLive.elt.addEventListener(“ended”, videoEnded,false);
howOld = createVideo(‘assets/howOld.mp4’);
howOld.hide();
howOld.elt.addEventListener(“ended”, videoEnded);
raceViolent = createVideo(‘assets/raceViolent.mp4’);
raceViolent.hide();
raceViolent.elt.addEventListener(“ended”, videoEnded);
sendPicture = createVideo(‘assets/sendPicture.mp4’);
sendPicture.hide();
sendPicture.elt.addEventListener(“ended”, videoEnded);
speakAnotherLanguage = createVideo(‘assets/speakAnotherLanguage.mp4’);
speakAnotherLanguage.hide();
speakAnotherLanguage.elt.addEventListener(“ended”, videoEnded);
beenHereBefore = createVideo(‘assets/beenHereBefore.mp4’);
beenHereBefore.hide();
beenHereBefore.elt.addEventListener(“ended”, videoEnded);
favoriteFood = createVideo(‘assets/favoriteFood.mp4’);
favoriteFood.hide();
favoriteFood.elt.addEventListener(“ended”, videoEnded,false);
yourName = createVideo(‘assets/yourName.mp4’);
yourName.hide();
yourName.elt.addEventListener(“ended”, videoEnded);
whatShipMade = createVideo(‘assets/whatShipMade.mp4’);
whatShipMade.hide();
whatShipMade.elt.addEventListener(“ended”, videoEnded);
randomOne = createVideo(‘assets/randomOne.mp4’);
randomOne.hide();
randomOne.elt.addEventListener(“ended”, videoEnded);
unspeakables = createVideo(‘assets/unspeakables.mp4’);
unspeakables.hide();
unspeakables.elt.addEventListener(“ended”, videoEnded);
sex = createVideo(‘assets/sex.mp4’);
sex.hide();
sex.elt.addEventListener(“ended”, videoEnded);
}
function videoEnded() {
console.log(“Video Ended”);
// videoEnd = true;
// recognition.start();
// print(“starting recognition”);
// background(0);
// image(stillImage, 0, 0);
//this.src = this.src;
//this.removeEventListener(‘ended’);
screenSeq = 2;
recognition.start();
print(“starting recognition”);
//var videoPlayed = videoPlayed +1;
}

// function startRecognition() {
// }

// function playNextVideo(videoName) {
// videoName.play();// this is not doing anything yet

// currentVideo = videoName;
// recognition.stop();
// }

// recognition.onresult = function(event) {
// var interim_transcript = ”;

// for (var i = event.resultIndex; i < event.results.length; ++i) {
// if (event.results[i].isFinal) {
// final_transcript += event.results[i][0].transcript;
// } else {
// interim_transcript += event.results[i][0].transcript;
// }
// }
// var w = split(final_transcript, ” “);
// for (var i = 0; i < w.length; i++) {
// print(w[i]);
// words.push(w[i]);
// }

// for (var i = 0; i < words.length; i++) {
// if (words[i] == “meaning”) {
// background(255);
// screenSeq = 3;
// }
// }
// };

function handleSpeech(event) {

//console.log(event.results[0][0][“transcript”]);
recognition.onresult = handleSpeech;
var res = event.results[0][0].transcript;

//print(res);
// logic here to search for keywords and responds
//index++;

recognition.stop();
console.log(“Speech”);
var w = split(res, ” “);
res = ”;

for (var i = 0; i < w.length; i++) {
print(w[i]);
words.push(w[i]);
}
console.log(words.length);
var checked = false;

background(127);
for (var i = 0; i < words.length; i++) {

if (words[i] == “meaning”) {
background(0);
screenSeq = 3;
meaningOfLife.play();
words = [];
w = [];
return;
} else if (words[i] == “where”) {
background(0);
screenSeq = 4;
whereAreYou.play();
words = [];
w = [];
return;
} else if (words[i] == “carbon”) {
background(0);
screenSeq = 5;
carbonBased.play();
words = [];
w = [];
return;
} else if (words[i] == “music”) {
background(0);
screenSeq = 6;
music.play();
words = [];
w = [];
return;
} else if (words[i] == “live”) {
background(0);
screenSeq = 7;
howLongLive.play();
words = [];
w = [];
return;
} else if (words[i] == “old”) {
background(0);
screenSeq = 8;
howOld.play();
words = [];
w = [];
return;
} else if (words[i] == “violent”) {
background(0);
screenSeq = 9;
raceViolent.play();
words = [];
w = [];
return;
} else if (words[i] == “picture”) {
background(0);
sendPicture.play();
screenSeq = 10;
words = [];
w = [];
return;
} else if (words[i] == “language”) {
background(0);
screenSeq = 11;
speakAnotherLanguage.play();
words = [];
w = [];
return;
} else if (words[i] == “here”) {
background(0);
screenSeq = 12;
beenHereBefore.play();
words = [];
w = [];
return;
} else if (words[i] == “food”) {
background(0);
screenSeq = 13;
favoriteFood.play();
words = [];
w = [];
return;
} else if (words[i] == “name”) {
background(0);
screenSeq = 14;
yourName.play();
words = [];
w = [];
return;
} else if (words[i] == “ship”) {
background(0);
screenSeq = 15;
whatShipMade.play();
words = [];
w = [];
return;
} else if (words[i] == “unspeakable”) {
background(0);
screenSeq = 17;
unspeakables.play();
w = [];
return;
} else if (words[i] == “sex”) {
background(0);
screenSeq = 18;
sex.play();
words = [];
w = [];
return;

}

}
screenSeq = 16;
randomOne.play();
}

function keyPressed() {
if (key == ‘ ‘) {
videoEnded();
}

}
// function mousePressed() {
// if (mouseX > 640 && mouseX < 790 && mouseY > 490 && mouseY < 540) {
// recognitionStart();
// }
// }
// function recognitionStart() {
// print(“starting recognition”);
// recognition.start();

// screenSeq = 1;
// }

function mousePressed() {
if (mouseX > 640 && mouseX < 790 && mouseY > 490 && mouseY < 540) {
opening.play();
screenSeq = 1;
// print(“starting recognition”);
}

}
function setup() {
createCanvas(1200, 800);

recognition = new webkitSpeechRecognition();
recognition.continuous = true;
recognition.onresult = handleSpeech;
}
function draw() {

//videoEnded();

// sequence

switch(screenSeq) {
case 0:
displayScene01();
break;
case 1:
displayScene02();
break;
case 2:
displayScene03();
break;
case 3:
displayScene04();
break;
case 4:
displayScene05();
break;
case 5:
displayScene06();
break;
case 6:
displayScene07();
break;
case 7:
displayScene08();
break;
case 8:
displayScene09();
break;
case 9:
displayScene10();
break;
case 10:
displayScene11();
break;
case 11:
displayScene12();
break;
case 12:
displayScene13();
break;
case 13:
displayScene14();
break;
case 14:
displayScene15();
break;
case 15:
displayScene16();
break;
case 16:
displayScene17();
break;
case 17:
displayScene18();
break;
case 18:
displayScene19();
break;

// case 9:
// displayRandomScene();
// break;
}

// // check words
// for (var i = 0; i < words.length; i++) {
// if (words[i] == “meaning”) {
// background(255);
// screenSeq = 2;
// } else if (words[i] == “look”) {
// background(255);
// screenSeq = 3;
// } else if (words[i] == “where”) {
// background(255);
// screenSeq = 4;
// }

// }
}

// function displayScene02() {
// // learn about “createVideo()”, it’s DOM !
// // movie.play();
// // movie.noLoop() //look up the exact wording
// draw background image – in code!

// }

function displayScene01() {
background(0);
fill(5, 17, 173);
rect(0, 0, 307, 207);

// logo and title of company
image(logo, 4, 4, 300, 200);
image(title, -10, 650, 1200, 200);

// menu in center of page
fill(100);
stroke(5, 17, 173);
rect(400, 250, 400, 300);

//Title of Dr. and log in boxes
textFont(“Orbitron”);
textSize(16);
fill(5, 17, 173);
text(“Dr. Andrej Raspinofski, PHD”, 410, 280);
textSize(14);
text(“Astrophysics and Linguistics”, 410, 300);
textSize(16);
text(“USER NAME”, 420, 372);
text(“PASSWORD”, 420, 441);
fill(200);
rect(540, 345, 200, 40);
rect(540, 415, 200, 40);

//log in name
fill(0);
stroke(0);
textFont(“Helvetica”);
text(“X8zp37VTi93ct639RP2”, 555, 370);
//password
ellipse(550, 435, 7, 7);
ellipse(560, 435, 7, 7);
ellipse(570, 435, 7, 7);
ellipse(580, 435, 7, 7);
ellipse(590, 435, 7, 7);
ellipse(600, 435, 7, 7);
ellipse(610, 435, 7, 7);
ellipse(620, 435, 7, 7);
ellipse(630, 435, 7, 7);
ellipse(640, 435, 7, 7);
ellipse(650, 435, 7, 7);
ellipse(660, 435, 7, 7);
ellipse(670, 435, 7, 7);
ellipse(680, 435, 7, 7);
ellipse(690, 435, 7, 7);
ellipse(700, 435, 7, 7);
ellipse(710, 435, 7, 7);
ellipse(720, 435, 7, 7);
ellipse(730, 435, 7, 7);
fill(5, 17, 173);

//log in button
rect(640, 490, 150, 50);
fill(255);
stroke(0);
textSize(“26”);
text(“LOGIN”, 675, 525);
}

function displayScene02() {
background(0);
image(opening, 0, 0);
//opening.play();

}

function displayScene03() {
background(0);
image(stillImage, 0, 0);
}

function displayScene04() {
//recognition.stop();
background(0);
image(meaningOfLife, 0, 0);
// meaningOfLife.play();
console.log(meaningOfLife.currentTime);
}

function displayScene05() {
//recognition.stop();
background(0);
image(whereAreYou, 0, 0);
// whereAreYou.play();

}

function displayScene06() {
// recognition.stop();
background(0);
image(carbonBased, 0, 0);
// carbonBased.play();
}

function displayScene07() {
//recognition.stop();
background(0);
image(music, 0, 0);
// music.play();

}

function displayScene08() {
//recognition.stop();
background(0);
image(howLongLive, 0, 0);
// howLongLive.play();

}

function displayScene09() {
//recognition.stop();
background(0);
image(howOld, 0, 0);
// howOld.play();

}

function displayScene10() {
//recognition.stop();
background(0);
image(raceViolent, 0, 0);
// raceViolent.play();

}
function displayScene11() {
//recognition.stop();
background(0);
image(sendPicture, 0, 0);
// sendPicture.play();

}

function displayScene12() {
//recognition.stop();
background(0);
image(speakAnotherLanguage, 0, 0);
// speakAnotherLanguage.play();

}

function displayScene13() {
//recognition.stop();
background(0);
image(beenHereBefore, 0, 0);
// beenHereBefore.play();

}

function displayScene14() {
//recognition.stop();
background(0);
image(favoriteFood, 0, 0);
// favoriteFood.play();
console.log(favoriteFood.currentTime);

}

function displayScene15() {
//recognition.stop();
background(0);
image(yourName, 0, 0);
// yourName.play();
console.log(yourName.currentTime);
}

function displayScene16() {
//recognition.stop();
background(0);
image(whatShipMade, 0, 0);
// whatShipMade.play();

}

function displayScene17() {
//recognition.stop();
background(0);
image(randomOne, 0, 0);
// whatShipMade.play();

}
function displayScene18() {
//recognition.stop();
background(0);
image(unspeakables, 0, 0);
// unspeakables.play();

}
function displayScene19 () {
background(0);
image(sex, 0, 0);
// sex.play();
}

To get the latest update of me and my works

>> <<