Btukfyl

I've been running a flask application with a celery worker and redis in three separated docker containers without any issue.

This is how I start it:

celery worker -A app.controller.engine.celery -l info --concurrency=2 --pool eventlet

Celery starts fine:

 -------------- celery@a828bd5b0089 v4.2.1 (windowlicker)

---- **** -----

--- * ***  * -- Linux-4.9.93-linuxkit-aufs-x86_64-with 2018-11-15 16:06:59

-- * - **** ---

- ** ---------- [config]

- ** ---------- .> app:         app.controller.engine:0x7f8ba4eb70b8

- ** ---------- .> transport:   redis://redis:6379/0

- ** ---------- .> results:     redis://redis:6379/1

- *** --- * --- .> concurrency: 2 (eventlet)

-- ******* ---- .> task events: ON

--- ***** -----

 -------------- [queues]

                .> celery           exchange=celery(direct) key=celery





[tasks]

  . app.controller.engine.do



INFO:engineio:Server initialized for eventlet.

INFO:engineio:Server initialized for threading.

[2018-11-15 15:44:34,301: INFO/MainProcess] Connected to redis://redis:6379/0

[2018-11-15 15:44:34,321: INFO/MainProcess] mingle: searching for neighbors

[2018-11-15 15:44:35,358: INFO/MainProcess] mingle: all alone

[2018-11-15 15:44:35,396: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.

[2018-11-15 15:44:35,415: INFO/MainProcess] celery@12af03844cd0 ready.

But when starting a task , it suddenly crashes:

[2018-11-15 15:02:03,224: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'float' object has no attribute 'items'",)

Traceback (most recent call last):

  File "/app/env/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start

    self.blueprint.start(self)

  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start

    step.start(parent)

  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start

    return self.obj.start()

  File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start

    blueprint.start(self)

  File "/app/env/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start

    step.start(parent)

  File "/app/env/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start

    c.loop(*c.loop_args())

  File "/app/env/lib/python3.6/site-packages/celery/worker/loops.py", line 121, in synloop

    connection.drain_events(timeout=2.0)

  File "/app/env/lib/python3.6/site-packages/kombu/connection.py", line 301, in drain_events

    return self.transport.drain_events(self.connection, **kwargs)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 963, in drain_events

    get(self._deliver, timeout=timeout)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 359, in get

    ret = self.handle_event(fileno, event)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 341, in handle_event

    return self.on_readable(fileno), self

  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 337, in on_readable

    chan.handlers[type]()

  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 724, in _brpop_read

    self.connection._deliver(loads(bytes_to_str(item)), dest)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 983, in _deliver

    callback(message)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 632, in _callback

    self.qos.append(message, message.delivery_tag)

  File "/app/env/lib/python3.6/site-packages/kombu/transport/redis.py", line 149, in append

    pipe.zadd(self.unacked_index_key, time(), delivery_tag) 

  File "/app/env/lib/python3.6/site-packages/redis/client.py", line 2263, in zadd

    for pair in iteritems(mapping):

  File "/app/env/lib/python3.6/site-packages/redis/_compat.py", line 123, in iteritems

    return iter(x.items())

AttributeError: 'float' object has no attribute 'items'

I don't see any reference to my code in the stacktrace.

This is the task method:

@celery.task()

def do(module_name, json_input):

    logger.info('____ Running _________________________')

    logger.info('testing ***********************')

    #modules.run(module_name, json_input)

I commented the call to my actual code just to verify I didn't mess it up with my modules code.. but it's crashing anyways.

Anybody has an idea what could be wrong or how to debug it properly?

Thanks a lot

The problem was that the my celery container downloaded a new release of the redis package with pip at build time, eventually with a bug or a serious change that crashes when celery tries to use it when connecting to redis. This new redis library is the 3.0.0 and it was released today.

To fix the issue you have to specify the previous version in the requirements.txt (which is 2.10.6).

And actually, specifying package versions in the requirements.txt is a good practice that prevents this type of problems.

I ran into this same issue today. hopefully this gets resolved shortly in an update by Celery. I reverted back to a previous version as well.

For anyone who use the docker-airflow, I used the folk of puckel/docker-airflow
Just need to modify the dockerfile by adding

   && pip install 'redis>=2.10.5,<3'

replace celery[redis] with only celery, by adding celery in apache-airflow built-in module i.e.,

  && pip install apache-airflow[crypto,celery,postgres,hive,jdbc,mysql]==$AIRFLOW_VERSION 

Update:
They also change the docker-airflow dockerfile recently to fix this bug, see here

Not enough rep yet for a comment:

This issue is tracked by the celery folks in:
https://github.com/celery/celery/issues/5175

Leading to this PR:
https://github.com/celery/celery/pull/5176

Which has been merged and reverted. Because some other fix landed. I'm not sure if this fixed it. It is no longer relevant for me.