Questions[connector](server)

I'm experiencing a strange situation.

My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is:

1. The Fetcher class would try to fetch an offset (e.g. 158473047).
2. It would get a "Fetch offset 158473047 is out of range for partition", then log the error and would reset the offset to ZERO.
3. When I search for that offset in the __consumer_offsets, it is there. So it's not a case that my offset was removed from the __consumer_offsets topic.
4. Due to this, my Kafka connector re-process a few hundred million messages.

Kafka Connect S3 log:

2020-05-11 05:18:41,372] INFO WorkerSinkTask{id=datalake-m3-to-s3-TEST-bulk-0} Committing offsets asynchronously using sequence number 16703: {M3.TEST-BULK-0=OffsetAndMetadata{offset=158473001, leaderEpoch=null, metadata=''}} (org.apache.kafka.connect.runtime.WorkerSinkTask:345)

[2020-05-11 05:18:42,020] INFO Opening record writer for: M3BULK/M3.TEST-BULK/year=2017/month=05/day=15/hour=18/M3.TEST-BULK+0+0158473001.snappy.parquet (io.confluent.connect.s3.format.parquet.ParquetRecordWriterProvider:69)

[2020-05-11 05:18:42,033] INFO [Consumer clientId=connector-consumer-datalake-m3-to-s3-TEST-bulk-0, groupId=connect-datalake-m3-to-s3-TEST-bulk] Fetch offset 158473047 is out of range for partition M3.TEST-BULK-0, resetting offset (org.apache.kafka.clients.consumer.internals.Fetcher:1256)

[2020-05-11 05:18:42,035] INFO [Consumer clientId=connector-consumer-datalake-m3-to-s3-TEST-bulk-0, groupId=connect-datalake-m3-to-s3-TEST-bulk] Resetting offset for partition M3.TEST-BULK-0 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState:385)

[2020-05-11 05:18:42,174] INFO Opening record writer for: M3BULK/M3.TEST-BULK/year=1970/month=01/day=01/hour=00/M3.TEST-BULK+0+0000000000.snappy.parquet (io.confluent.connect.s3.format.parquet.ParquetRecordWriterProvider:69)

Cluster Components:

• Kafka Build: confluent-community-2.12
• Installation Guide: https://docs.confluent.io/5.4.2/installation/installing_cp/rhel-centos.html
• Kafka node count: 3 (CentOS 7, 4 Cores, 16 GB RAM, 250 GB SSD each)

Kafka Broker server.properties content:

auto.create.topics.enable=true
broker.id=1
compression.type=producer
confluent.support.metrics.enable=false
controlled.shutdown.enable=true
delete.topic.enable=true
group.initial.rebalance.delay.ms=3000
default.replication.factor=2
group.max.session.timeout.ms=900000
log.retention.check.interval.ms=60000
log.retention.hours=24
log.segment.bytes=67108864
log.segment.delete.delay.ms=1000
num.io.threads=8
num.network.threads=6
num.partitions=1
num.recovery.threads.per.data.dir=4
offsets.topic.replication.factor=3
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
socket.send.buffer.bytes=102400
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3
unclean.leader.election.enable=false
zookeeper.connection.timeout.ms=6000
log.dirs=/data/kafka-data
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://KAFKA01.CONFLUENT.LOCAL:9092
zookeeper.connect=KAFKA01.CONFLUENT.LOCAL:2181,KAFKA02.CONFLUENT.LOCAL:2181,KAFKA03.CONFLUENT.LOCAL:2181

Important Points:

1. Cluster is NOT idle.
2. Server time is correct.
3. The offsets.retention.minute property is using the default value, which is 10080 minutes.

I really appreciate your help on this matter. Thank you!

I installed Azure AD Connector, followed the directions for an express install. When the sync occurred, everything was sync to include Service Accounts. All I want is active users that are in certain OU's. How do I remove what was synced and sync what I need? We have been using O365 for some time and the online account was updated (which was ok). I was reading this post https://azure.microsoft.com/en-us/documentation/articles/active-directory-aadconnectsync-configure-filtering/ but it came back with a Stopped Deletion Threashold. Did not want to proceed if it was going to delete everything fom O365.

Any help or suggestions?

I have problems sending emails outside organization using OWA (Exchange 2010 installed on SBS 2010, domain is domain.local, local DNS). Web-server and email server are hosted remotely, I am using pop connector to grab emails to Exchange 2010.

I have created two send connector, one that handles internal emails and one for external email. The problem is connector for emails over internet, which is configured like:

Addres Space SMTP, *, 1

Network Route email throught the following route host Smart host authentication: Basic authentication over TLS + username, password

If I don't use smart host I get the following error 451 4.4.0 Primary target IP address responded with: “454 4.7.5 Certificate validation failure.”

If I use smart host when using OWA outside of organization (different ISP), message gets stucked in a queue without being sent.

What am I doing wrong when configuring send connector?

Having a heck of a time trying to isolate an issue on a new server install:

I've installed MySQL, unixODBC and the MySQL Connector for ODBC, setup odbc.ini and odbcinst.ini as I have on my other 32-bit systems, but having an issue with applications (except PHP) receiving garbled data after a query.

For control purposes, here is what a MySQL direct query returns:

mysql> select id,username from users limit 3;
+------+----------+
| id   | username |
+------+----------+
|  786 | 101010   |
|  587 | 1234     |
| 1124 | 123456   |
+------+----------+
3 rows in set (0.00 sec)

mysql>

The same command in isql via ODBC returns this garbled result, only the first piece of data gets returned:

SQL> select id,username from users limit 3;
+-----------+---------------------------------+
| id        | username                        |
+-----------+---------------------------------+
| 786       |
|
|
+-----------+---------------------------------+
SQLRowCount returns 3
3 rows fetched
SQL>

My applications like isql are not able to return more than one piece of data. Strangely PHP scripts seem to work fine, but unfortunately my applications are not PHP based. Here is formatted PHP output:

ID: 786  Username:  101010
ID: 587  Username:  1234
ID: 1124  Username:  123456

odbc.ini

[site-db]
Description     = MySQL ODBC Site DB
Driver          = MySQL
User            = root
Password        = password
Database        = test
Option          = 3
Socket          = /var/lib/mysql/mysql.sock

odbcinst.ini

[MySQL]
DRIVER          = /usr/lib64/libmyodbc3.so
UsageCount              = 1

System Info:

OS: CentOs 5.7 x86_64
MySQL: Server version: 5.5.20-log MySQL Community Server (GPL) by Remi x86_64
MySQL Connector: v3.51.30-0 x86_64
unixODBC: 2.2.11 x86_64

If anybody can make some suggestions as to how I might be solved this issue, it would be greatly appreciated.

I just purchased an HP Microserver I was planning to use with FreeNAS. It needed more RAM, so I tried to follow the instructions to pull out the system board draw to access the DIMM slots. Doing so requires removing all of the cables connected to the board, including a "Mini-SAS" cable. The cable looks like it has an oversized USB-style connector with a metal clip to lock it into place similar to an RJ-45 connector's plastic clip. Try as I might, I couldn't get it to unlock, so I called HP tech support and they had me tug on it a good deal more still to no avail.

I'm thinking maybe the system shipped with a damaged connector, based on the HP video about disconnecting the cable. Has anyone worked with one of these systems? Can you comment on how hard it should be to disconnect that cable?

(Now the system won't start up and HP advised me to do an exchange, which I intend to do, but I found myself wondering if I'd have the same problem with the next unit.)