Initial release of Spycloud - #10608

Untested

request(
  "GET",
  state.url + "/breach/catalog?" + {
    ?"since": (state.?want_more.orValue(false) ? state.?cursor.first_published_date : state.?cursor.last_published_date).as(date,
      date.hasValue() ?
        optional.of([(timestamp(date.Value()) - duration("24h")).format(time_layout.DateTime)])
      :
        optional.none()
    ),
    ?"cursor": state.?want_more.orValue(false) ? [state.pagination_token] : optional.none(),
  }.format_query()
).with({
    "Header": {
      "x-api-key": [state.api_key],
      "User-Agent": ["ElasticSearch/0.1.0"],
    },
}).do_request().as(resp, resp.StatusCode == 200 ?
  bytes(resp.Body).decode_json().as(body,
    {
      "events": (has(body.results) && body.results.size() > 0) ?
        body.results.map(e, {"message": e.encode_json()})
      :
        [{"message": body.encode_json()}],
      "cursor": {
        ?"last_published_date": (has(body.results) && body.results.size() > 0) ?
          optional.of(([?state.?cursor.last_published_date] + body.results.map(e, e.spycloud_publish_date)).max())
        :
          state.?cursor.last_published_date,
        ?"first_published_date": (has(state.?cursor.first_published_date) && has(body.results) && body.results.size() > 0) ?
          (
            (has(body.cursor) && body.cursor != "" && state.want_more) ?
              state.cursor.first_published_date
            :
              state.cursor.last_published_date
          )
        :
          optional.none(),
      },
      "pagination_token": body.?cursor.orValue(""),
      "want_more": body.?cursor.orValue("") != "",
      "api_key": state.api_key,
    }
  )
:
  {
    "events": {
      "error": {
        "code": string(resp.StatusCode),
        "id": string(resp.Status),
        "message": string(resp.Body),
      },
    },
    "want_more": false,
    "api_key": state.api_key,
  }
)

Did you try the code above and find that it does not work? If so, what was the error?

Similar comments here.

I still don't think this is a good idea; it gives the wrong indication to the server who we are (we are not an Elasticsearch instance). Also the "S" should be lowercase.

https://pkg.go.dev/github.com/elastic/mito/lib#hdr-Global_Variables and https://pkg.go.dev/time#pkg-constants. But the constants are just strings and work the same way that Go time formats work, so if one does not exist, you can just use the literal, "2006-01-02".

?

Similar comments below.

Similar comments apply here.

Is this documented as being a required parameter? i.e. if the parameter were not present would we get the same behaviour?

Did you try the code above and find that it does not work? If so, what was the error?

What is the format of spycloud_publish_date? I assume this is a date format. "2006-01-02"? While this is not strictly a timestamp (it's a string), it does always sort the same way as the corresponding timestamp, so it should be ok to leave as a string. Is this a correct assessment?

What format are the timestamps here? If they are not lexically sortable with the same ordering as the chronological sort this is not correct. See #10608 (comment).

Same here.

Add to device ecs

As per developer (confirmed with Spycloud team), the values for the breach_catalog datastream are numeric, representing the actual count. Hence ECS mapping is not possible. Same for comments #10608 (comment) and #10608 (comment)

cc: @muskan-crest

Is this 24h look-back offset to prevent loss of data due to the low temporal resolution of the query parameter?

Data is generated daily at 12:00 AM in SpyCloud. For example, on July 31st, data will be created at 12:00 AM. If the API is accessed on August 1st at 3:00 PM, the operation labeled "now - 24h" will retrieve data starting from 3:00 PM on July 31st. Consequently, data from 12:00 AM to 2:59 PM on July 31st will be excluded and not retrieved. To prevent data loss, we implement a 24-hour look-back offset.

If the API is accessed on August 1st at 3:00 PM, the operation labeled "now - 24h" will retrieve data starting from 3:00 PM on July 31st.

Why? The "since" value does not get anything about the time of day; the format is "2006-01-02" which means that the query would be 2024-08-01. If you offset backwards by 24 hours, you will get 2024-07-31. If the query indicates to the API that it should be from the start of the day, then this offset is not needed, if it means from the end of the day, then it is.

Why are we using a fallback of null here? If it is to indicate that it should not be used, let's use an optional field.

Why does this differ from the case in the other main conditional branch where the fallback is to now - duration(state.initial_interval)? Note that putting that look-back deep in the program makes me queazy; I have been working on another integration that was doing this and I think that it was the cause of needless duplicate queries. Why is the look-back there?

Please fix the indentation here.

?
/cc @kcreddy

Agree with hide_sensitive as the parameter name instead of hide_sensitive_true.

Do we want to use a more searchable value?

If the API is accessed on August 1st at 3:00 PM, the operation labeled "now - 24h" will retrieve data starting from 3:00 PM on July 31st.

Why? The "since" value does not get anything about the time of day; the format is "2006-01-02" which means that the query would be 2024-08-01. If you offset backwards by 24 hours, you will get 2024-07-31. If the query indicates to the API that it should be from the start of the day, then this offset is not needed, if it means from the end of the day, then it is.

Same question here. I am not convinced that this is correct.

This operation is a requirement from the customer rather than from us. Since it has already been confirmed and tested by the customer, we cannot make any changes.

This one is less of a concern, but I'm still dubious about it.

Please fix the indentation of this block.

Please fix the indentation of this block.

Also, the resp.Body may be empty, so we synthesise a message in that case. Example here. This applies to the other cases here as well.