Any idea how to scrape city information?

Question

Hey guys! I'm working on a location application which users can choose the city where they live. Do you have any good ideas or source to get all the cities information?

chandu aa · Accepted Answer

1. You can scape city names from webpage like https://www.britannica.com/topic/list-of-cities-and-towns-in-the-United-States-2023068

Use jquery code like below: 
cities = $("div > a.md-crosslink").map(function(){ return $(this).text();}).get().join(',')

2. OR use google places autocomplete? https://developers.google.com/places/web-service/autocomplete

Shun Yamada · Answer

@chandu_aa Thanks so much for helpful info.
I have decided to use Google Places ApI :)

Answer

Construct the search results page URL from TripAdvisor – Tripadvisor has complex URL for the search results page of each locality. For example here is the one for Boston https://www.tripadvisor.com/Hotels-g60745-Boston_Massachusetts-Hotels.html. We’ll have to construct this URL manually to scrape results from that page. We do that by getting this URL from Tripadvisor autocomplete API.
Download HTML of the search result page using Python Requests – Quite easy, once you have the URL. We use python requests to download the entire HTML of this page.
Parse the page using LXML – LXML lets you navigate the HTML Tree Structure using Xpaths. We have predefined the XPaths for the details we need in the code.
Save the data to a CSV file

Any idea how to scrape city information?

Replies