Argthtjtr

I'm using Rselenium for scrapping. For this, I have installed java and JDK's, chromedriver, selenium server standalone and the headless browser phantomjs in my VM instance of Google Cloud.

I need to catch the text of the first rating:

remDr <- remoteDriver(browserName = 'chrome', port = 4444L)

remDr$open()

remDr$setWindowSize(1280L, 1024L)

remDr$navigate("https://www.ratebeer.com/reviews/sullerica-1561/294423")

text_post = remDr$findElements("xpath",'//*[@id="root"]/div/div[2]/div/div[2]/div[2]/div/div[1]/div[3]/div/div[2]/div[1]/div/div[2]/div/div[1]/div/div/div[1]')



text_post

## list()

Finally text_post is empty.

However, If I test the same script on my local laptop with RSelenium, chrome browser and the same XPath, it's a success!

What's going on?

Is it due to using phantomjs?

Thanks in advance.

sessionInfo()



R version 3.4.4 (2018-03-15)

Platform: x86_64-pc-linux-gnu (64-bit)

Running under: Ubuntu 16.04.5 LTS

As per the HTML you can use the xpath as:

//div[@id="root"]//span[contains(.,'20')]//following::div[contains(@class,'LinesEllipsis')]

Note: As the elements are dynamically generated elements you have to induce WebDriverWait for the Elements to be visible.

You don't need a heavyweight, third-party dependency. That site uses graphql POST requests under the hood in asynchronous XHR requests to retrieve the data. You can see it if you open Developer Tools and

I did a "Copy POST Data" (usually the same or rly similar context menu item in all browsers) and un-minimized the graphql query in the Response tab just to show you what it is and to also, perhaps, make it easier for you to see the query and augment it on your own (what I just said is out of scope for "but what about…" follow on questions in comments; please file a new question if you want help with that).

'[

    {

        "operationName": "beer",

        "query": "query beer($beerId: ID!) {n  info: beer(id: $beerId) {n    idn    namen    __typenamen  }n}n",

        "variables": {

            "beerId": "294423"

        }

    },

    {

        "operationName": "beer",

        "query": "query beer($beerId: ID!) {n  info: beer(id: $beerId) {n    idn    namen    styleScoren    overallScoren    averageRatingn    ratingCountn    __typenamen  }n}n",

        "variables": {

            "beerId": "294423"

        }

    },

    {

        "operationName": "beerReviews",

        "query": "query beerReviews($beerId: ID!, $authorId: ID, $order: ReviewOrder, $after: ID) {n  beerReviewsArr: beerReviews(beerId: $beerId, authorId: $authorId, order: $order, after: $after) {n    items {n      ...ReviewItemn      __typenamen    }n    totalCountn    lastn    __typenamen  }n}nnfragment ReviewItem on Review {n  idn  commentn  scoren  scores {n    appearancen    aroman    flavorn    mouthfeeln    overalln    __typenamen  }n  author {n    idn    usernamen    reviewCountn    __typenamen  }n  checkin {n    idn    place {n      idn      namen      cityn      state {n        idn        namen        __typenamen      }n      country {n        idn        namen        __typenamen      }n      __typenamen    }n    __typenamen  }n  servedInn  likeCountn  likedByMen  createdAtn  updatedAtn  __typenamen}n",

        "variables": {

            "beerId": "294423",

            "first": 7,

            "order": "RECENT"

        }

    }

]' -> graphql_query

We will need to scrunch that back into one line for the API call (which I do with the gsub() below. We also need to manually specify the content type and ensure httr does not try to mangle the body data by setting the encoding to raw:

httr::POST(

  url = "https://beta.ratebeer.com/v1/api/graphql/",

  httr::content_type("application/json"),

  encode = "raw",

  body = gsub("n", " ", graphql_query),

  httr::verbose()

) -> res

Now we have a structured, but heavily nested, list with your ifo in it. Pretty sure you're after items below:

str(httr::content(res), 4)

## List of 3

##  $ :List of 1

##   ..$ data:List of 1

##   .. ..$ info:List of 3

##   .. .. ..$ id        : chr "294423"

##   .. .. ..$ name      : chr "Sullerica 1561"

##   .. .. ..$ __typename: chr "Beer"

##  $ :List of 1

##   ..$ data:List of 1

##   .. ..$ info:List of 7

##   .. .. ..$ id           : chr "294423"

##   .. .. ..$ name         : chr "Sullerica 1561"

##   .. .. ..$ styleScore   : num 35.1

##   .. .. ..$ overallScore : num 51.8

##   .. .. ..$ averageRating: num 3.25

##   .. .. ..$ ratingCount  : int 21

##   .. .. ..$ __typename   : chr "Beer"

##  $ :List of 1

##   ..$ data:List of 1

##   .. ..$ beerReviewsArr:List of 4

##   .. .. ..$ items     :List of 10

##   .. .. ..$ totalCount: int 21

##   .. .. ..$ last      : chr "7177326"

##   .. .. ..$ __typename: chr "ReviewList"

It does only have 10 out of 21 so scroll down in your browser window with Developer Tools open and look at the second POST request that gets made, see what parameters changed and now you will have an even better idea of how to access the site's back-end API vs have to scrape for content.