Category Archives: Node.js

How to solve Postman “Could not get any response” error

Trying to build a REST API can be really frustrating. You install Express.js with mongoose or mongodb, and then create a schema, just following the instructions, and you think everything should work. You set up Postman to test the API, hit the API, and Postman stays at “sending request..” for a while. You wait… and wait… and then eventually you see the message “Could not get any response”.

Postman Could not get any response
Postman Could not get any response

Sometimes, it works fine to set everything up all at once. But, it can often happen that there are just too many moving parts. When that happens, and something breaks, it can be very difficult to figure out the point of failure.

As an example, suppose you have an Express router which handles a POST request like this using a mongoose model:

const router = require('express').Router();
// ... mongoose code here
router.post('/registerKitten', async (req, res) => {
    const fluffy = new Kitten({ name: 'fluffy' });
    try {
        await fluffy.save();
        res.json(fluffy);
    } catch(err) {
        res.status(400).send(err);
    }
})

If Postman times out when hitting the API, you can’t know whether the problem is in Express.js or Mongoose.

The first, simple thing you can do is add some console logging to see if your route is executed:

const router = require('express').Router();
// ... mongoose code here
router.post('/registerKitten', async (req, res) => {
    console.log("registerKitten");
    const fluffy = new Kitten({ name: 'fluffy' });
    try {
        console.log("awaiting save...");
        await fluffy.save();
        console.log("save is finished");
        res.json(fluffy);
    } catch(err) {
        res.status(400).send(err);
    }
})

That won’t tell you the whole story, however. If you see “awaiting save…” but don’t see “save is finished”, then doing this has helped you locate the problem – mongoose is hanging.

At this point, you may be better off writing a small Node.js test script which will just exercise the code related to mongoose, independently of Express. That way, you can debug the problem without having to deal with Express code or restart your Express server.

Here’s some sample code. You’ll have to adjust this to your own specific case, but the idea is to put this code into a script (test.js) and run it from the command line with node test.js:

"use strict";
const mongoose = require('mongoose');
const Kitten = require('./models/Kitten');

mongoose.connect(process.env.MONGO_DB_URL, { useNewUrlParser: true });
var db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));

// Set up some mock user data
const fluffy = new Kitten({ name: 'fluffy' });

// When the db is open, 'save' this data.
db.once('open', async function () {
    // we're connected!
    console.log("db opened... going to save the kitten");
    await fluffy.save();
    console.log("printing fluffy");
    console.log(fluffy);
    db.close();
});

// Print that db connection has been closed.
db.once('close', function () {
    console.log('close');
});

Notice that the code references an environment variable, process.env.MONGO_DB_URL. You’ll need to set that in your terminal. In Linux, you can do that by using the export command: export MONGO_DB_URL=mongodb://127.0.0.1:27017/myappdb.

If you found this interesting, click the subscribe button below! I write a new post about once a week.

PHP in_array or array_search vs JavaScript includes or indexOf

In my previous post, I’d created a bug in my PHP code by improper use of PHP’s array_search function. If you only write PHP, this kind of thing will probably never happen to you. I also write Java and JavaScript code, and at some point, your memory banks overflow, and you forget stuff!

JavaScript has a couple of different methods for searching through an array to find an item. Let’s use the array method includes to search for an item in JavaScript, just like I did in my PHP example.

var foods = new Array();
foods.push('ice cream');
foods.push('hamburger');
foods.push('brussels sprouts');

var do_important_things = function() { console.log('Important!'); };

if (foods.includes('ice cream')) {
    do_important_things();
}
...

If you print out foods, you can verify that ‘ice cream’ is located at index 0. The code foods.includes('ice cream') returns true because that string is found in the array. No confusion there – a boolean is returned, not an index.

If I’m doing a more complicated comparison, I might use JavaScript’s some, like this:

if (foods.some(function(el) { return el.toLowerCase() === 'ice cream'; })) {
    do_important_things();
}

The some method loops over all elements in the array until the callback method returns true, at which point it stops, and returns true. If no item is found which satisfies the condition in the callback, then false is returned. Again, since a boolean is returned and not an index, there’s no room for error.

There’s a JavaScript method which is the equivalent of the PHP array_search function: indexOf. It returns the index of the matching item, if found, and otherwise returns -1. Here’s an example:

if (foods.indexOf('ice cream')) {
    do_important_things();
}

Oops, this method of searching exhibits the same bug as array_search in PHP! In my opinion, the indexOf JavaScript method is more appropriately named – it makes it clear that you are going to get an index as a return value. So it makes it less likely that the all-too-human developer will create the bug that I demonstrated in my previous post.

Now, when researching this topic, I found that PHP has a function that is more appropriate than array_search for my use case. It’s the in_array function. It returns TRUE if the item being sought is found in the input array. That’s really what I wanted! Here’s the example code that I used in my previous post, only now it uses in_array:

$foods = Array();
$foods[] = 'ice cream';
$foods[] = 'hamburger';
$foods[] = 'brussels sprouts';

function do_important_things() {
    echo 'Important!';
};

if (in_array('ice cream', $foods)) {
    do_important_things();
}

In this example, do_important_things is called.

If you found this interesting, click the subscribe button below! I write a new post about once a week.

How to call the reddit REST API using Node.js – Part IV

This is the last of a 4-part series that describes how to call and use the reddit REST API using Node.js.

In Part I, I talked about using curl to get your access token, which gets you permission to use the reddit REST API.

In Part II, I used that access token to call reddit’s search API. But I was still using curl to do this. I have a very large string output from the API, and I don’t know about you, but I’m not keen on using Linux command line tools to process strings. Since I like JavaScript, I decided to move to Node.js for completing my work.

In Part III, I built a Node.js script which gets my access token from reddit. That script does pretty much what I’d been doing in part I, but it does it using Node.js.

Today, I’m going to complete the work by adding a reddit search API call to my Node.js script, and then using JavaScript’s handy string processing functionality to display the information that interests me.

Recall that I’m pretending to be responsible for Starbucks public relations, and I want to find out what’s being said about Starbucks at reddit, in case I need to do damage control!!

Here’s my new Node.js method which uses reddit’s search API to look for new entries which mention “Starbucks”:

const searchReddit = function (d) {
	const options = {
		hostname: "oauth.reddit.com",
		port: 443,
		path: "/r/all/search?q=Starbucks&sort=new",
		method: "GET",
		headers: {
			"Authorization": "Bearer " + d.access_token,
			"User-Agent": "fullStackOasis NewPostsScraper"
		}
	}

	const req = https.request(options, (res) => {
		// console.log(`statusCode: ${res.statusCode}`)
		let chunks = [];
		res.on('data', (d) => {
			// d is a Buffer object.
			chunks.push(d);
		}).on('end', () => {
			let result = Buffer.concat(chunks);
			let tmpResult = result.toString();
			try {
				let parsedObj = JSON.parse(tmpResult);
				// Print the string if you want to debug or prettify.
				// console.log(tmpResult);
				processSelfText(parsedObj);
			} catch (err) {
				console.log("There was an error!");
				console.log(err.stack);
				// I got an error, TypeError: Invalid data, chunk must be a string or buffer, not object
				// Also I got this, when I'd pushed d.toString to chunks:
				// TypeError: "list" argument must be an Array of Buffer or Uint8Array instances
				process.stderr.write(err);
			}
		});
	})

	req.on('error', (error) => {
		process.stderr.write(error);
	})

	req.end();	
};

As before, you do not have to understand this code in detail to see what’s going on. The input to my searchReddit function has the access_token which I’d previously obtained in Part III. This new code uses that access token to call the reddit search API, doing a search for “Starbucks”.

Buried in that code above is a call to a function processSelfText. I need that because it’s not helpful to have a giant wall of text displayed to me! I need to process this blob of data, and have the script display only the interesting parts.

My function processSelfText grabs the blob of JSON which was returned from reddit’s search API, and loops through it for all the individual reddit threads. It prints out a substring of the thread that contains the mention of “Starbucks”, and also prints out the reddit URL in case I want to read the whole thread. I can quickly skim through the results to see if the thread looks potentially harmful to Starbucks. If it does, then I can go to reddit to respond.

Here’s the string processing code:

const processSelfText = function (obj) {
	if (obj.data && obj.data.children && obj.data.children.length) {
		obj.data.children.forEach(function (item, n) {
			// data is an Object. It may have selftext property
			if (item.data) {
				console.log("Item #" + n);
				if (!item.data.selftext) {
					console.log("Only found a url, no text:");
					console.log(item.data.url);
				} else {
					console.log("Found url and text:");
					console.log(item.data.url);
					showSurroundingText(item.data.selftext);
				}
			}
		});
	}
}

/**
 * Process the input string to 
 * @param {*} str 
 */
const showSurroundingText = function (str) {
	let maxchars = 150;
	// Have to do a lowercase search.
	let found = str.toLowerCase().indexOf("starbucks");
	if (found > -1) {
		// See https://davidwalsh.name/remove-multiple-new-lines
		str = str.replace(/[\r]+/g, " ");
		str = str.replace(/[\n]+/g, " ");
		// If first argument is too large, it's okay, just returns front of string.
		// If second argument is too small, also okay.
		var substring = str.substring(found - maxchars, found + maxchars);
		// Remove the new lines.
		console.log("..." + substring + "...");
	}
};

I run the script from the command line, like this: node reputation-checker.js. I am using Node.js version 8.3, and Ubuntu 16.04, but I think this script will work for most other operating systems and platforms. This is what the output of my script looks like:

Item #0
Found url and text:
https://www.reddit.com/r/aznidentity/comments/cybh97/we_are_not_honorary_white_people_and_do_not/
 …tention span which many folks nowadays unfortunately do not.  I’ll order my morning coffee at an Asian-owned establishment rather than an overpriced Starbucks where it ends up tasting burnt anyway.  I could not care less for Mexican food or a Westerners version of Chinese food. Authentic Asian cuisi…
 Item #1
 Found url and text:
 https://www.reddit.com/r/exmormon/comments/cybd66/mom_why_dont_you_get_a_blessing_me_because_they/
 … of my skin alternates between burning and itching. Sometimes, I get both at once. Which is what happened yesterday. While I was waiting in line at Starbucks, talking to my mom on the phone.  My mom knows I left the church. And she knew I was at Starbucks. But none of that should matter. She’s livin…
 Item #2
 …

I could do more to refine this code – normally, I’d refactor it, and write some tests, and do more error handling, maybe automate it to send me an email periodically… but this is good enough for demonstration purposes. Here’s a link to the entire script, if you want to download it and mess around with it.

I hope you enjoyed this tutorial! Please feel free to use the “subscribe” form if you’d like to keep posted on updates to the “Full Stack Oasis” blog. I only post about once a week.

If you found this interesting, click the subscribe button below! I write a new post about once a week.

How to call the reddit REST API using Node.js – Part III

In Part I and Part II, I called the reddit REST API using the curl command line tool.

Now, I’m going to create a Node.js script that does this. Why create a script, when I can already do what I want with curl?

(1) I can more easily reuse the code that I’ve written. In Part II, I mentioned performing a search for the word “Starbucks” at reddit. If I wanted to do a different type of search, I could alter my script to search for something else instead.

(2) I can more easily execute the code that I’ve written. For example, I could run the script on a daily or hourly basis using a cron job, without having to do anything manually. “Set it and forget it” is awesome!

(3) I can more easily use the code that I’ve written. For example, I can bring in Node.js libraries to process the output of my script, and easily get from it what interests me.

So, now I’m going to move from curl to Node.js to do what I want, repeatedly, in an automated way.

Recall that I am pretending to be in charge of reputation management for Starbucks. I want to get recent comments that come up on reddit that mention Starbucks so I can quickly look for problems (or, hopefully, compliments!), and respond.

Below, I’ve shown just the Node.js code which can be used to retrieve my access token. Notice how much more complicated this is than the single curl command that I used previously! By the way, there are definitely simpler ways to do this using Node.js. I’m writing this example using just the built-in libraries that come with Node.js, which makes things a bit more complicated than they have to be.

const https = require('https');

let postData = "grant_type=password&username=my_user_name&password=MyExcellentPassword";
let username = "my_reddit_id";
let password = "my_reddit_secret";

/**
 * A method to get an access token to call reddit Search API
 */
const getAccessToken = function () {
	const options = {
		hostname: "www.reddit.com",
		port: 443,
		path: '/api/v1/access_token',
		method: 'POST',
		headers: {
			"Content-Type": "application/x-www-form-urlencoded",
			"Content-Length": postData.length,
			"Authorization": "Basic " + new Buffer(username + ":" + password).toString("base64"),
			"User-Agent": "my test bot 1.0"
		}
	}

	const req = https.request(options, (res) => {
		if (res.statusCode === 200) {
			let chunks = [];
			res.on('data', (d) => {
				/*
				* the output data has the format
				* {"access_token": "271295382352-tV_vIeKVRgq7Juh3iYHmW4oyT64",
				* "token_type": "bearer", "expires_in": 3600, "scope": "*"}
				* But d is a Buffer object, and has to be translated into an
			    * object at the end.
				*/
				chunks.push(d);
			}).on('end', () => {
				let result = Buffer.concat(chunks);
				let tmpResult = result.toString();
				try {
					let parsedObj = JSON.parse(tmpResult); // TODO do something with this Object, which contains my access token

				} catch (err) {
					process.stderr.write(err);
				}
			});
		} else {
			console.log("Received a statusCode " + res.statusCode);
		}
	});

	req.on('error', (error) => {
		process.stderr.write(error);
	})

	req.write(postData);
	req.end();
};

getAccessToken();

You don’t have to understand all this code in detail. If you skim it, you should get an idea what it’s doing. You are making an https request to reddit’s API. When a web browser makes an https request, there’s a lot of “stuff” going on under the hood. We have to code up some of that stuff here; that’s what the options object is for. Once the request is made to reddit, the information is returned in “chunks” of binary data over the network. Node.js waits for all of that data to get to us. When it’s finished, we use the built-in Buffer.concat method to concatenate all the binary data into one Buffer object, turn it into a JSON string, and parse that into an Object. The Object contains our access_token property. In my next post, we’ll use that to access the reddit API to search for recent posts about “Starbucks”.

If you found this interesting, click the subscribe button below! I write a new post about once a week.

How to call the reddit REST API using Node.js – Part II

In my previous post, I showed how easy it is to authenticate using the reddit API. Authenticating leaves me with an “access token”, which is just a JSON string which looks like this:

{"access_token": "261295382351-FBXDPTpUam35NR_UTJSXnjl5Pmd", "token_type": "bearer", "expires_in": 3600, "scope": "*"}

The access token gives me permission to use the reddit API. Now I can actually write an application that consumes the reddit API.

In order to motivate this demo, suppose you’re in charge of reputation management at Starbucks, and you want to be notified if anyone says anything about Starbucks at reddit. Fortunately, there’s a handy search API which can be used to search for the word Starbucks.

Here’s how you can use it to search for any mention of “Starbucks” at reddit:

curl -H "Authorization: bearer 261295382351-FBXDPTpUam35NR_UTJSXnjl5Pmd" --user-agent 'my test bot 1.0' -A "my app identifier" https://oauth.reddit.com/r/all/search?q=Starbucks

This returns a list of 25 results. If you check the search API that I linked to above, you’ll find that the default number of results (“limit”) is 25. The documentation doesn’t mention how results are sorted, but it looks like they are sorted by “relevance”, by default. We’d rather get them by “new”, to get the most recent posts. So let’s add that to the query string.

curl -H "Authorization: bearer 261295382351-FBXDPTpUam35NR_UTJSXnjl5Pmd" --user-agent 'my test bot 1.0' -A "my app identifier" https://oauth.reddit.com/r/all/search?q=Starbucks&sort=new

Now, your output is looking pretty hairy – a giant string of JSON text. You can take a look at this output in an online JSON “prettifier”, if you want to get a better idea of what’s being returned. This will be easier if your output is in a file. You can redirect the output of curl to a file. Just make sure that you put your URL in quotes, because the ampersand is interpreted by the shell to mean “start this command in the background”, which is not what you want. Do this:

curl -H "Authorization: bearer 261295382351-FBXDPTpUam35NR_UTJSXnjl5Pmd" --user-agent 'my test bot 1.0' -A "my app identifier" "https://oauth.reddit.com/r/all/search?q=Starbucks&sort=new" > data.json

Notice that so far, all I’ve done is use curl, a command line app, to call the reddit API! Now that I have a pretty good idea of how I’m going to be using the reddit API, I will start writing my Node.js application. I’ll dive into that next week.

If you found this interesting, click the subscribe button below! I write a new post about once a week.

How to call the reddit REST API using Node.js – Part I

One common programming task is to call a REST API. So it’s a good thing to know how to do this! In this demo, I’ll write a short Node.js program which “consumes” the reddit REST API. I’ll be doing this using a Linux environment (Ubuntu 16.04).

First, I had to sign up for a developer account at reddit. I was already a member of reddit, so I just used my own information to create the “app”. The reddit API access page is the place to go for signing up. There’s a link at the bottom which has the text “Read the full API terms and sign up for usage” – click that, and follow the instructions.

Next, go to your “Developed applications” page. Your “id” is found at the top left, a random string under the words “personal use script”. Click the “edit” link, below, and you’ll find your “secret”, another long string which should not be shared!

Using the reddit API requires authentication. Reddit needs to be able to know who is using their API, and to place limits on it, in case of abuse. So I need to figure out how to do this first.

Now that I’ve got an “id” and a “secret”, I can follow the instructions for authentication using OAuth2.

Let’s test the authentication process using curl, a command line tool which lets you “get” a web page from the command line.

curl -X POST -d 'grant_type=password&username=my_user_name&password=MyExcellentPassword' --user 'my_reddit_id:my_reddit_secret' https://www.reddit.com/api/v1/access_token

Copying and pasting that won’t work for you! Make sure to replace the strings my_user_name and MyExcellentPassword shown above with your own reddit sign-in information, and use your own id and secret as referenced above, when using curl.

After I did this, I saw the following output:

{"message": "Too Many Requests", "error": 429}

Oops. It turns out you need to set a user agent header to avoid this. Curl has a specific option, --user-agent, to let you do this. I just used a fake user-agent for my demo purposes: “my test bot 1.0”.

curl -X POST -d 'grant_type=password&username=my_user_name&password=MyExcellentPassword' --user 'my_reddit_id:my_reddit_secret' --user-agent 'my test bot 1.0' https://www.reddit.com/api/v1/access_token

Success! My output from curl looks like this:

{"access_token": "261295382351-FBXDPTpUam35NR_UTJSXnjl5Pmd", "token_type": "bearer", "expires_in": 3600, "scope": "*"}

This token will expire shortly… but that doesn’t matter. Now that I’ve got the authentication part working, I can work on the application, itself. I’ll cover that in my next blog post.

If you found this interesting, click the subscribe button below! I write a new post about once a week.