🏡 Deuce - Making House Hunting Easier For Couples pt2
12 min read

🏡 Deuce - Making House Hunting Easier For Couples pt2

How I learned to code whilst creating some of the many ideas I've come up with over the years

This is a follow on from my first ever post explaining how I came up with and designed Deuce.

2️⃣ Deuce

I didn’t name this idea in the last post and still didn’t even come up with one until after I’d built my next project. It took me almost a week after to find a name. Deuce. Deuce stands for two on dice or playing cards, and this idea is all about making life easier for two people house-hunting together. It’s not amazing, but it’ll do!


Summary

For those who skipped the first post in this series, here’s a snapshot summary of the problem I’m trying to solve::

💔 The problem: When searching for a property to buy with others, everyone involved needs to agree on buying the same property. There is no easy, digital way to identify 'ideal' properties as individuals and then compare them with those others buying the property have found. This means the home search process takes longer.

💡 The solution: A product that sits over the top of property portals that aggregates the saved property lists of those property portals, compares them and presents a list of properties that matched between both users and a list of properties which the user needs to review from the other user's saved lists.

🚧 Limitations: No front end development; the solution will work; it just won't look pretty doing it. I'll also be using one property portal to limit the volume of work.


Getting started

In case you missed it, I'm a coding noob. I dabbled with React.js for about a week back in 2019 and toiled with some Shopify and WordPress themes over the years but have never done anything serious. So as any noob would do, I enlisted Google for help and asked how to learn to code. The languages I came across the most were JavaScript and Python. I settled on using JavaScript because I found a free 1hr course on Udemy by Robin Haney.

Thankfully, I didn’t need to set up a code editor as I’d gone through the hassle back in 2019 when I tried to learn React.js, but the course helped me get to grips with the JavaScript syntax and some core concepts like variables and functions. Although, even after completing the course, the reality was I still hadn’t actually learnt anything; I’d just partially memorised it.

Variable: A variable is used to keep track of values in JavaScript.

Function: A function is a reusable block of code that groups together a sequence of code to perform a specific task.


🧰 The kit I'm using

Language: JavaScript
Code editor: Visual Studio Code
Environment: Node.js
Libraries: Puppeteer

🔬The initial research

Regardless of what language I was using to develop Deuce, I knew I had to find a way to interact with a webpage through code to even have a chance with building this thing. Not knowing if it was possible, I went on a hunt.

The first google search took me on a bit of a goose chase, but after almost an hour and 20 mins, I discovered Puppeteer.

This YouTube video by Aaron Jack helped me discover Puppeteer and what it could do, so I followed along, downloaded the Puppeteer library, and built a tool to scrape product information from Amazon. So far, so good. Copying someone is surprisingly easy.

Library: Libraries are a bunch of code that anyone can use. When you use a library,  it becomes a dependency as your project/code relies on this library to function. Libraries are there to simplify what you need to build and save you time.


🤡 What is Puppeteer?

Puppeteer is a library that enables you to control a headless browser through your code. The headless browser is a version of Chrome that can is hidden behind the scenes. Google created puppeteer to automate tests and tasks in Chrome, so it lets you do pretty much anything using code that you would normally be able to do manually, e.g. click on a link, type some text, scroll up and down. This was great because when you boil down what Deuce needs to do, it just needs to make a  few clicks, do a bit of typing and a whole lot of copying on-screen text.

More information on Puppeteer here:

So what did I do to set it up?

Enabling Puppeteer

The first thing I needed to do was install the library, and this was a case of typing npm i puppeteer in the terminal (more on this another time). The second thing was importing the puppeteer library into my code; to do that, I wrote this line of code at the top.

const puppeteer = require('puppeteer');

Starting a headless browser using Puppeteer

Next up, I had to work out how to actually start the headless browser and below is the next bit of code I entered. The ({ headless : false }) bit of the code determines whether the headless browser (basically chrome) is hidden or not, I had this set to false, so I could see the browser and see what was going on.

const browser = await puppeteer.launch({ headless : false });
const page = await browser.newPage();

Making Puppeteer look like a person

On the advice of Aaron Jack, the next thing I added was a user agent string. This was to make Puppeteer look like a legit computer using the web and disguise that it’s, in fact, a robot. At this point, I wasn't sure if I needed it, but as you will see later, this was actually a smart move. To get the user agent, simply google search "what is my user agent" and tada!

page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36');

Next was adding the code that would actually take you to a website. This was:

await page.goto('<https://www.rightmove.co.uk/user/shortlist.html>');

▶️Getting started and logging into a users Rightmove account

Now I need to work out how to scrape the saved properties for two users, compare them and spit out the matches. To do this, I'd need to somehow log into two users accounts. Low and behold, StackOverflow came through and gave me the answer.

I managed to work out a way to log into a users account by using the below code. This code essentially identifies the components on a page and tells the headless browser to do something. For example, line one tells the browser to look for the input field with an ID of email and enter the text 'abc'. The 3rd line is the same, except that it's a click, and the headless browser is looking for something with a class of mrm-button. To find the ID and class of an element in Chrome, right-click on your webpage, go to inspect and click on the thing you need the details for.

via GIPHY

I did have issues with clicking the log in button because I kept copying too much of the class information from the element on chrome, but I got over that, and I now have access to the users saved list of properties.

await page.type('input[name=email]', 'abc', {delay: 10});
await page.type('input[name=password]', '123', {delay: 10});
await page.click('.mrm-button',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'})

I'm in! I have the first users account at my beck and call. The challenge now is trying to retrieve the links for each saved property. Unfortunately, Rightmove doesn’t make this easy because there are no identifiers assigned to the list items for each card, and the same class name is used for Every. Single. One. this made working through them like a list impossible (at least for me right now...). I managed to find a workaround by completely ignoring the list and pulling the link information from the .sc-jbKcbu class assigned to the image on each property card. It's a crude way to solve the problem without actually solving it.

I spent a couple of hours trying to work out how to extract the information from the <a></a> (hyperlink) component, and I managed to build a robot that retrieves the user's list of saved properties and stores them as an array called propertyLinks. I'll later use these links to scrape all the details for each property. It seems strange, but this is such a small bit of code that seems to do something so complex, at least, what I think is complex.

const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));

Below is what the propertyLinks array looked like after the code was run.

[ '<https://www.rightmove.co.uk/properties/90775462>',
  '<https://www.rightmove.co.uk/properties/90775462>',
  '<https://www.rightmove.co.uk/properties/90443221>',
  '<https://www.rightmove.co.uk/properties/90443221>' ]

I noticed that there were duplicated links, so I set out to de-dupe it. I used the following code to store the unique links in an aptly named new array called uniquePropertyLinks. This not only appeases my OCD but will help to save compute time down the line when I want to extract the key property information for each link. I credit this snippet of knowledge to the information I found on the JavaScript Tutorial here.

let uniquePropertyLinks = [...new Set(propertyLinks)];

Now that I can retrieve an array of links for all the properties in a user's saved property list, I need to work out how to repeat it for a second user to identify the matches between the two users saved property lists.


⏏️ Logging user one out

To switch to the second user, I first needed to log the first user out.  When I first attempted to do this, I tried to find the button to be clicked from the Rightmove saved list page but couldn't find one, so I decided I'd need to navigate to a page with a logout button. At the time, this was https://www.rightmove.co.uk/myrightmove.html.

Once on this page, I needed to click the logout button, but unfortunately, the class assigned to the button was also used in four other places meaning Puppeteer wouldn't know which one to click or click the wrong one. So on cue, Google came to the rescue, and I researched some options.

I settled on using the inner text 'Log out' to identify where the button was on the page and then click it. I borrowed some code to do it, and honestly, I don't understand how it works (it's beyond my capabilities at this stage). The source was GitHub from a developer called Tokland.

<https://gist.github.com/tokland/d3bae3b6d3c1576d8700405829bbdb52>

//Code to locate text and enable it to be clicked
const escapeXpathString = str => {
  const splitedQuotes = str.replace(/'/g, `', "'", '`);
  return `concat('${splitedQuotes}', '')`;
};

const clickByText = async (page, text) => {
  const escapedText = escapeXpathString(text);
  const linkHandlers = await page.$x(`//a[contains(text(), ${escapedText})]`);
  
  if (linkHandlers.length > 0) {
    await linkHandlers[0].click();
  } else {
    throw new Error(`Link not found: ${text}`);
  }
};

//Logout
  await page.goto('<https://www.rightmove.co.uk/myrightmove.html>');
  await clickByText(page, 'Log out');
  await page.waitForNavigation({waitUntil: 'load'});
  console.log('Signed out');

So now I can log in as a user, create an array of links that the user has in their saved property list and then log out. Next, I wanted to repeat this for a second user, and I would need to move away from using static text for the login details to do it.

🥺 Embarrassing addition... I made this harder than it needed to be...

It actually turns out, from the saved properties list page, there is a logout button. I didn't look hard enough! What adds insult to injury is that on the saved properties list page, the logout button is the only button that uses the class sc-kAzzGY. I could have gotten away using a simple one-liner of code adapted from what I used to click the login button. I'm not too fussed, though; I enjoyed the mini-challenge, trying to do something new, then reaching the high of finding a solution. If the bulkier code I previously used help me to understand a little more about puppeteer and what's possible with it, then I'm cool with that.

await page.click('.sc-kAzzGY',{delay: 10});

🏃‍♂️ Running the code for a second user

After thinking about how I would approach switching between the users, the only way I could do it was by duplicating the code with the username and password variables being different. Still, I knew this wasn't an efficient way of writing the code. I deferred to Stackoverflow for support (A superuser changed my question to be boring, obvs personality is not allowed.). A user named pavelsaman came in with a comprehensive response giving some decent guidance without spelling out everything I'd need to do. This is the kind of answer I like as it means I can learn instead of copy and pasting, which I'm coming to learn is a core competency of being a developer.

I did, however, come up with a potential solution to my problem whilst waiting for a response via StackOverflow. The inspiration for this was the code in the below lesson review challenge from Codecademy, and it made me realise you can store data as properties in an object.

This was the code I created, intended to use an object called activeUser with properties activeUser.email and activeUser.password to replace the existing code and then use a new function called moveToNextUser(); to update the activeUser to the second user once the property links for the first user have been retrieved.

let userAEmail = 'abc';
let userAPassword = '123';
let userBEmail = 'def';
let userBPassword= '456';

const activeUser = { email: userAEmail, password: userAPassword };

const moveToNextUser = activeUser => {
activeUser.email = userBEmail;
activeUser.password = userBPassword;
};

moveToNextUser(activeUser);

With the above in mind, I added the properties of the object (email and password) now need to the login code created earlier, and it looked like this:

await page.type('input[name=email]', **activeUser.email**, {delay: 10});
await page.type('input[name=password]', **activeUser.password**, {delay: 10});
await page.click('.mrm-button',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'})

I tried to implement this in the code, but I couldn't get it to work, so I deferred back to the advice I got from pavelsaman on StackOverflow and changed the approach. This led me to some new phraseology, 'data structure' and a new way of structuring data I'd not yet come across. What pavelsaman was suggesting made complete sense; by structuring my data in a variable, it wasn't possible to do what I needed, so a new data structure was required. I spent some time reading up on objects and properties here to get to grips with what I was dealing with and, to my surprise, found a great article by Tania Rascia published on digital ocean that used some lord of the rings references. A light-hearted way to make some boring code interesting.

The below code was the object-based structure pavelsaman recommended.

const users = {
  userA: {
      email: 'abc',
      password: '123',
  },
  userB: {
      email: 'def',
      password: '456',
  },
};

Once this was in place, I then created a for-in loop to access the login details in the object and rotate the details used between the users on each iteration of the loop. The below code was what I used to check the loop was working. The output of both users details confirmed it was a success.

for (const user in users) {
console.log(users[user])
};

//Result
[ 'abc','123'], 
[ 'def','456']

There is a problem, though. I need to find a way to access the email and password properties independently and not as an array like ['abc', '123']. After some googling, I found that to access properties, you need to use .property. With this in mind, to access the email property of each user, I need to use users[user].email with users[users] identifying the user dynamically in the for-in loop and .email being the property I'm accessing.  Similarly, to access the user's password, it's users[user].password.  I credit my newfound knowledge on accessing the properties of objects to this blog post by Dimitri Pavlutin.

The updated login code now looks like this:

await page.type('input[name=email]', **users[user].email**, {delay: 10});
await page.type('input[name=password]', **users[user].password**, {delay: 10});
await page.click('.mrm-button',{delay: 10});
await page.waitForNavigation({waitUntil: 'load'})
console.log('Signed in');

I can now pass the object users into the function to retrieve the saved property lists, create an array and repeat with a loop for each user in the object. Which, after a long Saturday afternoon writing code, means I can:

  1. Log into the Rightmove account of userA
  2. Create an array of saved property links called uniquePropertyLinks for userA
  3. Log out of the userA account
  4. Log into the Rightmove account of userB
  5. Create an array of saved property links called uniquePropertyLinks for userB
  6. Log out of the userB account

🏁 At this point, I decided to stop for the day and finish up tomorrow; the code I ended with is shown below. In my next post, I'll complete the build and present the final product.

const puppeteer = require('puppeteer');

//Object storing user credentials
let userAEmail = 'abc';
let userAPassword = '123';
let userBEmail = 'def';
let userBPassword = '456';

const users = {
  userA: {
      email: userAEmail,
      password: userAPassword,
  },
  userB: {
      email: userBEmail,
      password: userBPassword,
  },
};

//Function to retrieve users saved list of properties
async function retrieveUserSavedList(users) {
  try {

    //Load broswer
    const browser = await puppeteer.launch({ headless : false });
    const page = await browser.newPage();
    page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36');
  
    for (const user in users) {

      //Go to Rightmove saved list
      await page.goto('<https://www.rightmove.co.uk/user/shortlist.html>', {waitUntil: 'networkidle2'});
      await page.waitForSelector('.mrm-button');
      console.log('Page loaded');

      //User log in
      await page.type('input[name=email]', users[user].email, {delay: 10});
      await page.type('input[name=password]', users[user].password, {delay: 10});
      await page.click('.mrm-button',{delay: 10});
      await page.waitForNavigation({waitUntil: 'load'})
      console.log('Signed in');

      //Wait for user list to load
      const propertyList = await page.$$('.title');
      console.log(propertyList.length);

      //Collecting saved property links and de-duping into an array
      const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));
      let uniquePropertyLinks = [...new Set(propertyLinks)];
      console.log(uniquePropertyLinks);

      //Sign out
			await page.click('.sc-kAzzGY',{delay: 10});
      await page.waitForNavigation({waitUntil: 'load'});
      console.log('Signed out');
    };

} catch (err) {
    console.log('Error retrieve user saved list - ', err.message);
  } 
  
};

//Run the code
retrieveUserSavedList(users);