Skip to content

stream wrapper functionality #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kisaga wants to merge 14 commits into
base: main
Choose a base branch
from
Draft

stream wrapper functionality #27

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
0386e59
added Parameter object for searchStream and sampleStream methods in T…
kisaga Jun 13, 2022
5eb7ddd
created TwitterStream and moved executor from examples
kisaga Jun 14, 2022
2d016d3
checking permorfance for different number of threads
kisaga Jun 15, 2022
09f86fc
minor improvements
kisaga Jun 17, 2022
bab514d
switched from InputStream and BufferedReader to okio BufferedSource
kisaga Jun 17, 2022
9d28fe1
reduced duplication for closing executor service and switched to sing…
kisaga Jun 17, 2022
39e3261
added disconnect functionality when stream is empty for 20 secs and s…
kisaga Jun 17, 2022
f7f5c2d
adding logging instead of system.out or .err
kisaga Jun 18, 2022
00bfeea
added tests and licenses
kisaga Jun 18, 2022
c1c27c6
added retries
kisaga Jun 18, 2022
658decb
Merge remote-tracking branch 'upstream/main' into feature/add-stream-…
kisaga Jun 18, 2022
7086775
added docs and some minor changes
kisaga Jun 18, 2022
96f1b74
java date time instead of millis vars for empty stream timeout mechanism
kisaga Jun 19, 2022
18068c8
switched back to gson for deserialization thread
kisaga Jun 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added retries
  • Loading branch information
@kisaga
kisaga committed Jun 18, 2022
commit c1c27c65d6cca65a147ceafb53e6faaf45506479
19 changes: 17 additions & 2 deletions src/main/java/com/twitter/clientlib/stream/TwitterStream.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,18 @@ public class TwitterStream {

private TweetsStreamExecutor executor;

private int retries;

public TwitterStream() {
init();
}

public TwitterStream(int retries) {
this.retries = retries;
init();
}

private void init() {
initBasePath();
tweets.setClient(apiClient);
}
Expand All @@ -55,6 +66,10 @@ public void setTwitterCredentials(TwitterCredentialsBearer credentials) {
apiClient.setTwitterCredentials(credentials);
}

public void setRetries(int retries) {
this.retries = retries;
}

public void addListener(TweetsStreamListener listener) {
listeners.add(listener);
}
Expand All @@ -65,7 +80,7 @@ public void removeListener(TweetsStreamListener listener) {

public void sampleStream(StreamQueryParameters streamParameters) {
try {
BufferedSource streamResult = tweets.sampleStream(streamParameters);
BufferedSource streamResult = tweets.sampleStream(retries == 0 ? 1 : retries, streamParameters);
executor = new TweetsStreamExecutor(streamResult);
listeners.forEach(executor::addListener);
executor.start();
Expand All @@ -78,7 +93,7 @@ public void sampleStream(StreamQueryParameters streamParameters) {
/* for this error twitter indicates that to implement a reconnection mechanism
* see: https://developer.twitter.com/en/docs/twitter-api/tweets/volume-streams/integrate/handling-disconnections
*/
throw new TooManyRequestsException("Too many requests. Service responded with 429 status code");
throw new TooManyRequestsException("Too many requests. Service responded with 429 status code. Consider setting 'retries' or increasing its value");
}
throw new StreamException("An exception occurred during stream execution ",e);
}
Expand Down
1 change: 1 addition & 0 deletions src/test/java/com/twitter/clientlib/stream/TwitterStreamTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ class TwitterStreamTest extends ApiTester {
public static void beforeAll() {
twitterStream = new TwitterStream();
twitterStream.setTwitterCredentials(new TwitterCredentialsBearer(System.getenv("TWITTER_BEARER_TOKEN")));
twitterStream.setRetries(4);
}

@BeforeEach
Expand Down