🏑 Deuce - making house hunting easier for couples pt3
12 min read

🏑 Deuce - making house hunting easier for couples pt3

It's now Sunday, and after an intense morning the day before building my first product, Deuce, I was ready to get back into it. I had a tight timeline, with only 3 hours to build the remainder of Deuce as I was due to meet up with some friends for a socially distanced pint in the cold.

To recap where I left off on Saturday, the code I built enables Deuce to:

  1. Log into the Rightmove account of userA
  2. Create an array of saved property links called uniquePropertyLinks for userA
  3. Log out of the userA account
  4. Log into the Rightmove account of userB
  5. Create an array of saved property links called uniquePropertyLinks for userB
  6. Log out of the userB account

Sunday's challenge was to finish the code, which meant storing the properties collected in the current code and working out which properties in both users saved lists matched. I was excited for this session because it was less about using Puppeteer and all about learning new JavaScript methods and functions.


πŸ—‚οΈ Storing the saved property lists of each user

So, storing the saved property links for each user. I need to store this information for each user, so I can then retrieve it, compare the lists of links and flag the matches between the two. At the moment, my code stores the property links for the first user to an array called uniquePropertyLinks and then the loop for the second user writes the property links for the second user to the same array replacing the first users' links. From now on, I will be calling the first user userA and the second user userB.

To get around this problem, I created a unique array for each user and stored the property links to the relevant users' array. I used a similar data structure to what I implemented for the user credentials, and it looked like this:

//Object storing users saved lists as arrays
const usersPropertyLinks = {};

usersPropertyLinks.userA = [];
usersPropertyLinks.userB = [];

I then stored the temporary information held in the uniquePropertyLinks array to the relevant array for each user. I decided to try using an IF function to do this and placed it after the existing code that de-dupes the propertyLinks array into the uniquePropertyLinks array.

On my first attempt, this looked like this:

//Add saved property links to an array for each user
if (users[user] === userA ) {
			  usersPropertyLinks[userA].push(uniquePropertyLinks);
			  console.log(usersPropertyLinks);
} else if (users[user] === userB ) {
        usersPropertyLinks[userA].push(uniquePropertyLinks);
        console.log(usersPropertyLinks);
} else {
        console.log('problem saving links to user array');
};

This code didn't work for several reasons, but let's break down what I was thinking.

I had to work out which user the loop was looking at, so with the code user[user] I'd used earlier to identify the right user credentials; I checked to see if it was userA or userB. This code looks like if (users[user] === userA ). This produces a boolean response (true or false), and if true, the code assigned for userA would trigger (i.e. writing the property links to an array for userA), but if false, it moved on to the next bit of the IF function and checked if it was userB. This looked like else if (users[user] === userB ) again, if true, the code assigned for userB would be triggered and write the property links to an array for userB. If both of these came up false, I had written an error message to capture it else { console.log('problem saving links to user array'); };.

So why wasn't the code working?

The first error message was 'Cannot read property 'userA' of undefined'. The error is tells me that it either can't read the value of userA or the value of userA was indeed undefined. I knew it wasn't the latter because userA has been defined in a couple of places, so I focused on finding out why it couldn't read the value of userA. After 30 mins of pouring over the code, I found a problem. I wasn't passing the usersPropertyLinks object into the function... so I fixed it:

retrieveUserSavedList(users, **usersPropertyLinks**);

I'm now passing usersPropertyLinks into the function, but I'm still getting an error this time slightly different 'userA is not defined.' The change I made this time around was improving the IF function. I made it more specific to remove userA from the equation and to hopefully solve the issue. The code now looked like this:

//Add saved property links to an array for each user
if (**users[user].email === userAEmail** ) {
			  usersPropertyLinks[userA].push(uniquePropertyLinks);
} else if (**users[user].email === userBEmail** ) {
        usersPropertyLinks[userA].push(uniquePropertyLinks);
} else {
        console.log('problem saving links to user array');
};

But it didn't work. Whilst it improved the code, it still didn't solve the problem. I eventually found that the error was being caused by how I'd written the code for identifying the properties of the usersPropertyLinks object. I'd used usersPropertyLinks[userA], but it should have been usersPropertyLinks.userA a simple syntax issue caused by my inexperience. I'd done this because I'd previously used users[user] for the user credentials object and assumed it would be the same here. The code now looked like this:

//Add saved property links to an array for each user
if (users[user].email === userAEmail ) {
        usersPropertyLinks.userA.push(uniquePropertyLinks);
} else if (users[user].email === userBEmail ) {
        usersPropertyLinks.userB.push(uniquePropertyLinks);
} else {
        console.log('problem saving links to user array');
};

And it worked. The code now updates the usersPropertyLinks object and when I log it to the console using console.log(usersPropertyLinks); it looks like this:

{ userA:
   [ [ '<https://www.rightmove.co.uk/properties/90775462>',
       '<https://www.rightmove.co.uk/properties/90443221>' ] ],
  userB:
   [ [ '<https://www.rightmove.co.uk/properties/90775462>',
	     '<https://www.rightmove.co.uk/properties/105563375>',
	     '<https://www.rightmove.co.uk/properties/78989052>',
	     '<https://www.rightmove.co.uk/properties/78112053>',
	     '<https://www.rightmove.co.uk/properties/103006508>',
	     '<https://www.rightmove.co.uk/properties/88509475>',
	     '<https://www.rightmove.co.uk/properties/96263414>'  ] ] }

At this point, it looks correct, but I had a gut feeling something was going on with the links being double wrapped with square brackets. I thought it would cause an issue down trying to compare the links in the arrays, so I resolved it by cleaning up my code.

Initially, the code that produced the double wrapped links looked like this:

//Collecting saved property links and de-duping into an array
      const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));
      let uniquePropertyLinks = [...new Set(propertyLinks)];

//Add saved property links to an array for each user
      if (users[user].email === userAEmail ) {
        usersPropertyLinks.userA.push(uniquePropertyLinks );
      } else if (users[user].email === userBEmail ) {
        usersPropertyLinks.userB.push(uniquePropertyLinks );
      } else {
        console.log('problem saving links to user array');
      };

I removed the extra array wrapper by simply removing the uniquePropertyLinks array and pushed the de-duped list of links straight to the array for the relevant user. The improved code now looked like this:

//Collecting saved property links and de-duping into an array
      const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));

//Add saved property links to an array for each user
      if (users[user].email === userAEmail ) {
        usersPropertyLinks.userA.push(...new Set(propertyLinks));
      } else if (users[user].email === userBEmail ) {
        usersPropertyLinks.userB.push(...new Set(propertyLinks));
      } else {
        console.log('problem saving links to user array');
      };

The output now looks like this:

{ userA:
   [ '<https://www.rightmove.co.uk/properties/90775462>',
     '<https://www.rightmove.co.uk/properties/90443221>' ],
  userB:
   [ '<https://www.rightmove.co.uk/properties/90775462>',
     '<https://www.rightmove.co.uk/properties/105563375>',
     '<https://www.rightmove.co.uk/properties/78989052>',
     '<https://www.rightmove.co.uk/properties/78112053>',
     '<https://www.rightmove.co.uk/properties/103006508>',
     '<https://www.rightmove.co.uk/properties/88509475>',
     '<https://www.rightmove.co.uk/properties/96263414>' ] }

🍐 Comparing the saved property lists of each user

Next up comparing the saved property list of userA with userB to find the properties that match. I'm expecting only to see the link '<https://www.rightmove.co.uk/properties/90775462'> as none of the others matches. Whilst thinking about how to go about this, Rightmove caught on to me... every time I now ran the code, the following error came up when looping through the second user.

I had run the code well over 100 times, so I suspected that they're on to me using a robot, and it wasn't a problem on their end as they described. I went back to the user agent details I set up earlier and made a tweak to the Chrome bit, changing from Chrome/90.0.4430.93 to Chrome/90.0.4430.91. Surprise, surprise, it worked, and I can get back to work. I’ll probably need to come back to this down the line to randomise the number being used such that this doesn’t happen again.

On to comparing the property lists and work out the matches between two arrays. I started my research back at StackOverflow, where all the coding geniuses seem to live and found this question. It provided a comprehensive insight into how you can get the difference and similarities between the two arrays. The bit that was most important just now was the similarities referred to as the intersection in the answer.

I implemented this into my code as per the below. I was careful to make sure this code wasn't in the for-in loop that I'd been focusing on until now.

let matchingProperties = usersPropertyLinks.userA.filter(x => usersPropertyLinks.userB.includes(x));
    console.log(matchingProperties);

The result was as expected, just one property output:

[ '<https://www.rightmove.co.uk/properties/90775462>' ]

I managed to do this the first time around, so at this point, I was feeling like a real coder despite knowing that less than 5 minutes ago, I didn't know this function existed.


⛏️ Gathering the details for the shared saved properties

Now I had the link for the saved properties that matched between the users, I wanted to gather the key high-level information. The first step was to work out what information I needed from each property page. Thankfully this was easier than retrieving the links from the saved property lists, as the property pages are pretty well structured. I am assuming this is because Rightmove wants these pages to be indexed well by Google, but I'll be honest, I don't actually know why. This is what the code now looks like:

//Property name
const propertyName = await page.$eval('._2uQQ3SV0eMHL1P6t5ZDo2q', _2uQQ3SV0eMHL1P6t5ZDo2q => _2uQQ3SV0eMHL1P6t5ZDo2q.innerText);
  
//Property price
const propertyPrice = await page.$eval('._1gfnqJ3Vtd1z40MlC0MzXu', _1gfnqJ3Vtd1z40MlC0MzXu => _1gfnqJ3Vtd1z40MlC0MzXu.innerText);
  
//Property type
const propertyType = await page.$eval('._1fcftXUEbWfJOJzIUeIHKt', _1fcftXUEbWfJOJzIUeIHKt => _1fcftXUEbWfJOJzIUeIHKt.innerText);
 
//Property agent
const propertyAgent = await page.$eval('._2rTPddC0YvrcYaJHg9wfTP', _2rTPddC0YvrcYaJHg9wfTP => _2rTPddC0YvrcYaJHg9wfTP.href);

The page.$eval method essentially searches the page for the class I've defined and then, once it finds it, extracts the inner text, which is the information I am after. Using the property name as an example, the code searches the page for the class ._2uQQ3SV0eMHL1P6t5ZDo2q then returns the text associated with it.

Below is what this looks like on the Rightmove property listings, and in this example, the variable propertyName would be assigned the value 'Fairfax Drive, Westcliff-on-Sea'.

At the moment, I’ve only been able to get the robot to return the data for the first link in the propertyLinks array, so again I'll need to create a loop. I used a for-of loop this time Β to capture all the data for each of the links in the matchingProperties array. In practice, this loop will go to each property link in the matchingProperties array and then return the propertyName, propertyPrice, propertyType and propertyAgent as shown in the above code for each. This code looks like this:

for ( const propertyLink of matchingProperties) {
      await page.goto(propertyLink);
      const propertyName = await page.$eval('._2uQQ3SV0eMHL1P6t5ZDo2q', _2uQQ3SV0eMHL1P6t5ZDo2q => _2uQQ3SV0eMHL1P6t5ZDo2q.innerText);
      const propertyPrice = await page.$eval('._1gfnqJ3Vtd1z40MlC0MzXu', _1gfnqJ3Vtd1z40MlC0MzXu => _1gfnqJ3Vtd1z40MlC0MzXu.innerText);
      const propertyType = await page.$eval('._1fcftXUEbWfJOJzIUeIHKt', _1fcftXUEbWfJOJzIUeIHKt => _1fcftXUEbWfJOJzIUeIHKt.innerText);
      const propertyAgent = await page.$eval('._2rTPddC0YvrcYaJHg9wfTP', _2rTPddC0YvrcYaJHg9wfTP => _2rTPddC0YvrcYaJHg9wfTP.href);
			console.log(propertyLink, propertyName, propertyPrice, propertyType, propertyAgent);
    };

πŸ—ƒοΈ Storing the property details of each shared saved property

The next step is to save the results of the loops so I can retrieve them elsewhere, possibly use them to present information to the user. To do this, I used the same data structure that I used for the user credentials and the storing of each user's saved property links. The difference this time is that I needed the loop to add a new property to the object with the property details. Before, I had only used an object with pre-defined properties, i.e. user A and userB. With this in mind, I drafted what I thought would be the right way to structure the data, shown below and referred to my trusty colleague Google.

const matchingPropertyDetails = {};

matchingPropertyDetails.property1 = [ link: '', name: '', price: '', type: '', agent: '' ];
matchingPropertyDetails.property2 = [ link: '', name: '', price: '', type: '', agent: '' ];
matchingPropertyDetails.property..n = [ link: '', name: '', price: '', type: '', agent: '' ];

The code I came up with was:

//Object storing matching saved properties as arrays
const matchingPropertyDetails = {};

//Loop to collect details for the matching saved properties
for ( const propertyLink of matchingProperties) {
      await page.goto(propertyLink);
      await page.waitForSelector('._2uQQ3SV0eMHL1P6t5ZDo2q');

      //Property link
      matchingPropertyDetails.propertyLink.Link = propertyLink;
      //Property name
      const propertyName = await page.$eval('._2uQQ3SV0eMHL1P6t5ZDo2q', _2uQQ3SV0eMHL1P6t5ZDo2q => _2uQQ3SV0eMHL1P6t5ZDo2q.innerText);
      matchingPropertyDetails.propertyLink.name = propertyName;
      //Property price
      const propertyPrice = await page.$eval('._1gfnqJ3Vtd1z40MlC0MzXu', _1gfnqJ3Vtd1z40MlC0MzXu => _1gfnqJ3Vtd1z40MlC0MzXu.innerText);
      matchingPropertyDetails.propertyLink.price = propertyPrice;
      //Property type
      const propertyType = await page.$eval('._1fcftXUEbWfJOJzIUeIHKt', _1fcftXUEbWfJOJzIUeIHKt => _1fcftXUEbWfJOJzIUeIHKt.innerText);
      matchingPropertyDetails.propertyLink.type = propertyType;
      //Property agent
      const propertyAgent = await page.$eval('._2rTPddC0YvrcYaJHg9wfTP', _2rTPddC0YvrcYaJHg9wfTP => _2rTPddC0YvrcYaJHg9wfTP.href);
      matchingPropertyDetails.propertyLink.agent = propertyAgent;
    };

Unfortunately, this came up short and kept erroring. The message was 'Cannot set property 'name' of undefined', so I deferred once again to StackOverflow, this time to a question that is older than all the kids my girlfriend teaches.

In summary, the problem I had was that I was trying to assign a property to an object called matchingPropertyDetails.propertyLink where .propertyLink represents the URL for the saved property without first creating the object matchingPropertyDetails.propertyLink. I then ran the code once more and found that the objects were literally being named matchingPropertyDetails.propertyLink as opposed to matchingPropertyDetails.<https://ww>... this was an issue as it meant I couldn't dynamically add new matched saved properties to the object. This was a nice simple solution and just needed .propertyLink to be replaced with [propertyLink]. Once I updated these two problems, the code was working seamlessly and now looks like this:

//Object storing matching saved properties as arrays
const matchingPropertyDetails = {};

//Loop to collect details for the matching saved properties
    for ( const propertyLink of matchingProperties) {
      await page.goto(propertyLink);
      await page.waitForSelector('._2uQQ3SV0eMHL1P6t5ZDo2q');

      matchingPropertyDetails[propertyLink] = {};

      //Property link
      matchingPropertyDetails[propertyLink].Link = propertyLink;
      //Property name
      const propertyName = await page.$eval('._2uQQ3SV0eMHL1P6t5ZDo2q', _2uQQ3SV0eMHL1P6t5ZDo2q => _2uQQ3SV0eMHL1P6t5ZDo2q.innerText);
      matchingPropertyDetails[propertyLink].name = propertyName;
      //Property price
      const propertyPrice = await page.$eval('._1gfnqJ3Vtd1z40MlC0MzXu', _1gfnqJ3Vtd1z40MlC0MzXu => _1gfnqJ3Vtd1z40MlC0MzXu.innerText);
      matchingPropertyDetails[propertyLink].price = propertyPrice;
      //Property type
      const propertyType = await page.$eval('._1fcftXUEbWfJOJzIUeIHKt', _1fcftXUEbWfJOJzIUeIHKt => _1fcftXUEbWfJOJzIUeIHKt.innerText);
      matchingPropertyDetails[propertyLink].type = propertyType;
      //Property agent
      const propertyAgent = await page.$eval('._2rTPddC0YvrcYaJHg9wfTP', _2rTPddC0YvrcYaJHg9wfTP => _2rTPddC0YvrcYaJHg9wfTP.href);
      matchingPropertyDetails[propertyLink].agent = propertyAgent;
      console.log(matchingPropertyDetails);
    };

The final output looks like this:

{ '<https://www.rightmove.co.uk/properties/90775462>':
   { Link: '<https://www.rightmove.co.uk/properties/90775462>',
     name: 'Fairfax Drive, Westcliff-on-Sea',
     price: 'Β£350,000',
     type: 'End of Terrace',
     agent: '<https://www.rightmove.co.uk/estate-agents/agent/Bear-Estate-Agents/Southend-on-Sea-135017.html>' } };

πŸ“ Summarising the journey

At 9 minutes past midnight on Monday (technically now Tuesday), this now concludes my weekend journey to build a tool that would combine my partners and my house hunting lists to show the ones we have mutually saved.

As a new developer, I enjoyed the challenge and thrilled that I managed to build this with a tool (puppeteer) and a language that 3 days ago was completely alien to me. I came with some core concepts, an open mind and determination to build something that has been on my idea list for a couple of months and leave with a Deuce, a product that solves the problem and a whole lot of knowledge and experience I can take into the next project.

Below you'll find the final code along with a video of it in action. It's not a pretty solution and only works with one property portal. Maybe I'll come back to this at some point to build it out further.

On to the next project!


Deuce in action

The final code

const puppeteer = require('puppeteer');

//Object storing user credentials
let userAEmail = 'abc';
let userAPassword = '123';
let userBEmail = 'def';
let userBPassword = '456';

const users = {
  userA: {
      email: userAEmail,
      password: userAPassword,
  },
  userB: {
      email: userBEmail,
      password: userBPassword,
  },
};

//Object storing users saved lists as arrays
const usersPropertyLinks = {};

usersPropertyLinks.userA = [];
usersPropertyLinks.userB = [];

//Object storing matching saved properties as arrays
const matchingPropertyDetails = {};

//Function to retrieve users saved list of properties
async function retrieveUserSavedList(users, usersPropertyLinks, matchingPropertyDetails) {
  try {

    //Load broswer
    const browser = await puppeteer.launch({ headless : true });
    const page = await browser.newPage();
    page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.90 Safari/537.36');
  
    for (const user in users) {

      //Go to Rightmove saved list
      await page.goto('<https://www.rightmove.co.uk/user/shortlist.html>', {waitUntil: 'networkidle2'});
      await page.waitForSelector('.mrm-button');

      //User log in
      await page.type('input[name=email]', users[user].email, {delay: 10});
      await page.type('input[name=password]', users[user].password, {delay: 10});
      await page.click('.mrm-button',{delay: 10});
      await page.waitForNavigation({waitUntil: 'load'})
      console.log('Success: ' + users[user].email + ' logged in');

      //Collecting saved property links and de-duping into an array
      const propertyLinks = await page.evaluate(() => Array.from(document.querySelectorAll('.sc-jbKcbu'), e => e.href));

      //Add saved property links to an array for each user
      if (users[user].email === userAEmail ) {
        usersPropertyLinks.userA.push(...new Set(propertyLinks));
      } else if (users[user].email === userBEmail ) {
        usersPropertyLinks.userB.push(...new Set(propertyLinks));
      } else {
        console.log('problem saving links to user array');
      };

      //Sign out
      await page.click('.sc-kAzzGY',{delay: 10});
      await page.waitForNavigation({waitUntil: 'load'});
      console.log('Success: ' + users[user].email + ' logged out');
    };

    //Find matching saved properties between userA and userB
    let matchingProperties = usersPropertyLinks.userA.filter(x => usersPropertyLinks.userB.includes(x));

    //Loop to collect details for the matching saved properties
    for ( const propertyLink of matchingProperties) {
      await page.goto(propertyLink);
      await page.waitForSelector('._2uQQ3SV0eMHL1P6t5ZDo2q');

      matchingPropertyDetails[propertyLink] = {};

      //Property link
      matchingPropertyDetails[propertyLink].Link = propertyLink;
      //Property name
      const propertyName = await page.$eval('._2uQQ3SV0eMHL1P6t5ZDo2q', _2uQQ3SV0eMHL1P6t5ZDo2q => _2uQQ3SV0eMHL1P6t5ZDo2q.innerText);
      matchingPropertyDetails[propertyLink].name = propertyName;
      //Property price
      const propertyPrice = await page.$eval('._1gfnqJ3Vtd1z40MlC0MzXu', _1gfnqJ3Vtd1z40MlC0MzXu => _1gfnqJ3Vtd1z40MlC0MzXu.innerText);
      matchingPropertyDetails[propertyLink].price = propertyPrice;
      //Property type
      const propertyType = await page.$eval('._1fcftXUEbWfJOJzIUeIHKt', _1fcftXUEbWfJOJzIUeIHKt => _1fcftXUEbWfJOJzIUeIHKt.innerText);
      matchingPropertyDetails[propertyLink].type = propertyType;
      //Property agent
      const propertyAgent = await page.$eval('._2rTPddC0YvrcYaJHg9wfTP', _2rTPddC0YvrcYaJHg9wfTP => _2rTPddC0YvrcYaJHg9wfTP.href);
      matchingPropertyDetails[propertyLink].agent = propertyAgent;
      console.log(matchingPropertyDetails);
    };

    browser.close();

} catch (err) {
    console.log('Error retrieve user saved list - ', err.message);
  } 
  
};

//Run the code
retrieveUserSavedList(users, usersPropertyLinks, matchingPropertyDetails);